But they really should have been with Aerys the whole time. Rhaegar took Barristan and 2 other KG with him to the Trident where they all died except for Barristan. Then they left Jaime alone with Aerys in KL while the three of them went to the Tower.
Aerys was still alive, they should've ALL been with him. And if they had already decided to take orders from Rhaegar over Aerys, then they were already all Oathbreakers so their actions no longer reflect the code of the KG.