... The infinitive is not "to go," it is just "go." It is often found without "to," in constructions such as "Can we go to the store?" or "She'll go ape when she hears this." It's a single word that often stands by itself.
As I said, the "to" is analogous to the "have" in "have gone" -- a marker that's part of a certain grammatical inflection of a verb. There's no law that says you can't put words between "have" and "gone." "I have occasionally gone to that store" is perfectly valid....
My apologies for the double post; Christopher, you snuck in while I was writing.
There is no infinitive in those examples. Those are merely compound verbs formed with a main verb and an auxiliary. Split those all you want, according to "the rules." ("I will gladly support you."
But when we write "to + verb," now it's not a verb, now it is an infinitive, a verbal, functions as something else, bla, bla, bla, see post above. I am in full agreement that they should be deemed splittable.
But it is reality that in the past many people deemed it wrong to interrupt the to and the verb. The fact that it was considered wrong is not myth.