The Doctor was in love with Seven, even I could see that. And given enough time, I could totally see her returning his affections. And it's certainly not unheard of for a teacher to fall in love with a student (or the reverse). In fact I think Doctor/Seven would have been the most realistic of all the possible Seven pairings I can think of. In a way, they have more in common than a lot of people realize.
As for Harry? I honestly don't blame him for backing away. We guys are trained to believe that a woman's attentions are something we must work for...earn, even. Most of us would be totally clueless if she just threw herself at us without preamble. I'm sure Harry did consider that Seven was "innocent" and not somebody he should take advantage of, but also, he may have been thinking "What's the catch?"
However different the sexual mores of the 24th century might be, some things will never change. Harry does not strike me as the type who'd be into casual encounters. If he fancies Seven, he'd want to go about it the right way - with romance.