Good points. I can only guess if my theory holds true that it wasn't so much about the crew not knowing about it, but Janeway's personal ethics and rules for such a relationship.
One recurring theme in Voyager is that it's impossible to keep any secrets on a ship that small.
And if part of Janeway's ethical parameters for carrying on a secret (or semi-secret) romance with a subordinate involves the destruction of said subordinate's career... well, that's some pretty twisted ethics.