Tom called himself a "pilot" which is not true completely.
He was the department head in charge of the helm.
He didn't drive Voyager 24/7, and randos couldn't pick up the slack when he got sleepy or had something better to do.
"Department head" is why he got to sit in on mission briefings.
Say there are 20 people on Voyager who are qualified to fly the ship, then he has to figure out duty shifts and promotion tracks and reprimands based of performances. He also has to plot the course and replot the course and they get a more exact idea about what's ahead, and review how the trip matches up with what they anticipated, as well as make sure that the hardware is working fine, and that anyone who wants to upskill is given the classes they want, and receive credits for those classes, and their slice of the duty pie changes with regards to their new skill set.
That's a Lieutenant's job, not an Ensigns, who was displaced, promoted, dead or demoted when Janeway gave Tom his old job back.