Yeah, I have to think long and hard about putting 2 above 4 and sometimes it drops anyway. You're absolutely right -- it has more awesome episodes than the fourth year but that middle stretch is pretty painful to get through.
I would say "The Muse" is probably the worst episode of the fourth season but it's not as bad as four or more second-season outings.
That opening arc in the sixth season is terrific.
Everything you said about why the seventh season is at the top for me is exactly how I feel, except of course that those lesser elements don't quite bother me as much as they do most people. I'm also an Ezri fan. It's true that introducing a new cast member in the final year of a dramatic series can (and to some extent did) cause problems but I enjoyed her development enough to let it slide.