If Janeway was the love of his life why should he potentially spend 70 years pining after someone who won't give him the time of day? The crew were ging to be stranded for 70 years in universe, the fact it only took 7 years, well 7 is still a long time.
Maybe because, right the beginning of their journey (no matter that it ran 7 or 70 years), Janeway was very clear on one of her golden rules: she would NEVER pursue a romantic affair with a subordinate of her crew. In accepting to be her First Officer and serve her, Chakotay became de facto, one of her subordinates.
-> hey, if the Doctor understood the point, why Chakotay didn't?!
-> Chakotay who boasted to know Janeway, it is surprising that he doubted that she would keep her position on this point. He would have known to what extent Janeway could be stubborn when she put herself in it!
(It is certainly what I like most in this character, she has principles and she is held in it, even if she has to suffer consequences afterwards).
And then, there was "Resolutions". Even if after almost 3 years spent in the DQ, Chakotay was still interested in her and he clearly was, it seems to me that Janeway didn't share the same interest in him. Indeed, during the supposed 3 months spent on this unknown island, absolutly nothing happened between them and God knows how Chakotay tried to initiate a romance or something. Janeway was stayed stone-faced or almost from the beginning till the end, refusing to answer Chakotay's advances, even after having resolved to be infinitely cornered on this island. Maybe she would have eventually had a change of heart/mind after even several months but nothing is less safe!

.
-> Mulgrew uses to say that producers tried to make Janeway/Chakotay happen and I think that she refered to first seasons but, that it didn't work and Resolutions marked this "ended even before having begun" situation. And afterward, neither Mulgrew nor Beltran saw the interest in trying again (I guess that the cordial relationship between both actors already had the lead in the wings!

)