Why are americans so weird about men in speedos?

I like to wear boxer brief style swimsuits b.c they are comfortable, don't drag etc. I can't stand having baggy clothes on in the water. they fill with air bubbles. (girls imagine swimming in a skirt). Anyway, in america, there seem to be many people who get uncomfortable when a man wears a speedo. Why does anyone care? Aren't you supposed to be free to do your own thing?