Oh, I agree. And I think Janeway went about it the wrong way when she just led him on and then deleted his memories; full disclosure would have been more respectful. But, the Doctor should have understood that Voyager needed him, and if a certain memory set threatened his destruction, it needed to go. Just as he would have, if no alternative treatment existed, amputated a patient's limb.
This brings to a character assassination of the whole crew, in "Unforgettable" we're told that the whole ship can lose several weeks of its collective memory without anyone ever noticing that something is amiss. How can it be done to being with? I suppose they all have personal logs, all 150 of them. How can you alter all those logs in a period of three weeks while keeping them coherent, that doesn't make any sense. And that woman, she says "pheromones" prevent people from remembering her. How about the people who never met her, on a ship of that size it's very likely, but have heard of her from someone else? How can their memories be erased? Voyager has many many bullshit episodes like that, completely undefendable.