Follies and fouls of Rhaenyra. Let us count the ways.
Sleeping with a Kingsguard and losing her virtue before she's married. This would normally get a Kingsguard gelded and sent to the Wall. Bedding the princess is treason. Criston should've been put to death. This should have disinherited Rhaenyra from the succession. But because Viserys believes it was his brother and not a Kingsguard, he overlooks it.
Blowing off her suitors and being rude in another lord's castle (Storm's End/Baratheon), while meeting lords from the Riverlands (whose liege lords are the Tully's). This wasn't just about finding a husband, but also a meet and greet to the major houses in Westeros. Rhaenyra needed to charm and win these lords support if she is going to ascend the throne. Being a woman already puts her at a disadvantage. The fact that Viserys made the lords at court swear oath to Rhaenyra against their will, was another strike against her. When Rhaenyra laughs and jests at Jerrel Bracken's mocking insults to Willem Blackwood; while Willem is making a serious marriage proposal to the princess. When the duel between the Bracken and Blackwood starts, Rhaenyra just walks away and doesn't try to stop the violence. It certainly didn't endear her to the lords of the Riverlands or lord Baratheon. There are many houses who hold grudges and feuds against one another. If the future monarch can't be bothered to quell the violence among just two. Why should any lord support her?
Getting married and having 3 bastard children is a colossal error. In the books, Laenor isn't black, but he does have that white/silver blonde hair. It's obvious to everyone in the Red Keep and at court who the father of Rhaenyra's children are. Once again, Rhaenrya doesn't care because her daddy is king. Rhaenyra likely figured once she is queen, she can legitimize all her sons and put an end to any issue. What should have happened was Rhaenyra should've been exiled, along with her 3 children and Ser Strong put to death. Once again, Viserys chose not to do anything about it.
Visery's motive for stubbornly not removing Rhaenyra as heir is out of guilt for killing her mother. Rhaenyra is the last thing the king has of his late wife, Aemma. He wants to honor the memory of his wife and ensure Rhaenyra has something worthy. But Rhaenyra hasn't been acting like a lady, let alone a princess for more than a decade.