The point about the Kingsguard is a point I've been making for a while elsewhere. The Kingsguard serves the King. The King can extend Kingsguard protection to his family. But ultimately, the King comes first. Jaime recalls listening to Aerys II abuse his wife and being told by another KG (Darry I think) that they are supposed to protect the Queen from everyone except the King.
If Aerys II did not order those 3 Kingsguard to be at the Tower of Joy, then Hightower, Dayne, and Whent were all oathbreakers. They chose the prince over the king, which is not the purpose of the Kingsguard. If they're oathbreakers then Ned should hold them in contempt like he holds Jaime.