I've been trying to stress that point for years.
But lets just remember one thing.
He sent the Cardassias back.
But Neelix said that there's dozens of ships out there with the same sob story as Janeway.
How come Gul Evek got sent home yet Ransom was set adrift?
Of course some say the Vetar avoided the Caretakers grasp, but there's little proof one way or the other.
I was always under the impression that Evek and the Vetar weren't drug in and it was just some other Cardassian ship. Just the conclusion I drew from the facts available. Either way, there's no canon appearance for Evek after Caretaker anyways, so it's open ended. He could be back on Cardassia, he could be aimlessly wondering the Delta Quadrant.
Unless of course you buy into the theory that the Caretaker was working with the Cardassians to secretly plot against the Federation...

Moral of the story is consistency sucks! Just ask Voyager's writers.