Hello surrealmastermind and welcome to Artist Forum.
I read of a reknown fantasy artist who had the exact same trouble; he was working on a comic book series with a lot of beautiful, scantily-clad women and somehow (I forget) he no longer worked for that company and the next paying gig he got was drawing a mostly male superhero team! Of course, after drawing all these women for such a long time he was practically lost on how to draw men. What did he do?
He kept at it until he could draw men.
The same hand-eye acumen you have for drawing women is there to draw men or fruit or buildings or whatever. But you can't do it without doing it. Just commit to draw a bunch of screwy looking men until you get better at it; try referencing, copying/redrawing pictures of the kind of look of men you're generally aiming for. If the masters (Peter Paul Reubens, Da Vinci, Salvador Dali, etc.) didn't look down upon themselves for searching outside of themselves for inspiration development, why should any of the rest of us act coy about it?
You know the technicals: men's hips tend to be narrower than a woman of the same size, shoulders a little broader...overall a man's physique is more angular, chiseled, revealing more muscular detail. There's no royal road...but impatience and frustration can make a journey of any length more miserable than necessary.