There are two things that have to happen to bring equality to an unequal situation.
Someone must come up or someone must go down. Usually, I think, it's a little bit of both.
And I think this is what is terrifying to white heterosexual men (as well as all kinds of other people groups who have generally benefited from inequality).
It's not that anyone is worried about them being brought up, it's the worry of us going down.