I'm also getting some specks of rust which I've stopped with working them over with fine steel wool and, using a q-tip, applying some lightweight gun oil. (EEZX). But the oil film does attract dirt.
If the brackets show rust, then they're not aluminum; a magnet sticks to them, so they're ferrous. Therefore they can't be anodized. Maybe some sort of plating?
Powder coating might be an option. I wish they were made of stainless steel.