^Kovarian had spent much of her life believing the Doctor to be a bogeyman who would destroy the universe, because of her interpretation of the prophecy of Trenzalore. She was also closely affiliated with a group of religious fanatics, the Silence. It's entirely possible that she could've learned to hate him without him having to do anything to her personally. Hate is often based more on imagination and false beliefs, such as racial or sexual stereotypes or the propaganda of a warlike govenrment, than on genuine grievances. That's part of what makes hatred such a dangerous and destructive force.