Men’s buttons are on the right to make it easy to do up with one hand for right-handed people.
When buttons were invented, only the wealthy wore them. Wealthy women had maids who would do the buttons up for them, and so the buttons are on the left so that they are easier for someone facing you to do up.
Your head’s in the right place. A lady was expected to be dressed and undressed by servants. But even a non-aristocratic woman would want to dress the same as the courtly ladies, and not wear men’s clothes so women’s’ shirts got buttons on the left side, regardless of the wearer’s class and the idea just persisted through to the present.