Season 4 is doing really well so far, only a single episode out of 17 rated below 5.
Season 4 definitely is Voyager's highpoint for me
Season 4 is definitely my favorite season so far. I think that has a lot to do with 7 of 9, who is currently my favorite character on the show (and not just because she's hot
I hope that the high ratings continue.
EDIT: And to answer my original question, I think season 4 is where the show gets A LOT better. I'm finally excited to sit down and watch Voyager every night. There hasn't been a single bad episode since season 3.