I have watched it again, and no, they don't fall too fast. They fall as fast as people always do in movies. Virtually every fall an actor makes on screen is sped up to make it look far more violent than it actually was, because usually, the actor hasn't fallen at all, but has merely flopped like a fish and reacted as if he's fallen. Or was a stunt performer, and not the actor at all, and the footage has been sped up to not give you enough time to spot the difference. Likewise, they almost always speed up punches, and occasionally remove frames at the supposed moment of impact, to create a visual "jump" that creates a greater sense of violent impact, even though no actual impact occurred.
It's called "movie magic".