But we don't hear much about his attraction to Jadzia after season 1, do we? I suppose it's possible that he was in love with her after this point and didn't realise it until it was triggered years later, but given his infatuation with her during season 1, it seems like a stretch to think that he loved her but didn't know.
I've always taken it to be that at this early stage he was just infatuated with her based on her appearance, like you said. However, after he backed off a little and actually started to get to know her, he began to fall in love with her.
Given that I'm no ladies man, I might be talking out of my ass, but I've always thought that relationships start with an infatuation like we see with Bashir but then develop into something deeper.