I don't know if we can look too closely to the Targ interbreeding for an explanation of anything. IIRC, Martin has admitted that genetics aren't his strongpoint. In this continuity, all Baratheons have black hair and all Targs have purple eyes and silvery hair; it's not really like real-world genetics apply.
May seem a pedantic point but not ALL Targs have purple eyes and silver hair. Princess Rhaenys took after her Dornish mother not Rhaegar in appearance and Baelor Breakspear in The Hedge Knight also doesn't look traditionally Targ (or traditionally Valyrian). A minor point but an important one given how many times I've heard "But how can Jon be a Targaryen if he's got brown hair?"