We don't know exactly what piece of machinery did what in this process. In "Trouble with Tribbles", we get another example of people beaming from one Federation pad to another, and there, too, it's unclear whether the Enterprise
machinery or the K-7 machinery was doing the work, or whether both were involved. In contrast, in "Ultimate Computer" we see Scotty operating the sliders when people beam aboard from the local space station...
The dialogue in ST:TMP is ambiguous enough.
Cleary: "Transporter room, come in! Urgent! Redline on the transporters, Mister Scott!"
Scotty: "Transporter room, do not engage! Do not-"
Cleary: "It's too late. They're beaming now!"
Rand: "Do you read me, Starfleet? Override us. Pull them back!"
Kirk: "Give it to me. Starfleet, boost your matter gain, we need more signal! ...More signal!"
Scotty: "We're losing their pattern."
Rand: "Oh, no! They're forming!"
Kirk: "Starfleet, do you have them?"
Starfleet: "Enterprise... What we got back didn't live long... Fortunately."
What we learn is that Rand "engaged", which is not necessarily the same as her energizing the starship's machinery. Might be her simply signaling the go-ahead for Starfleet to energize. And it's not clear who Cleary is referring to when saying "They" are beaming - is it the transporter room, is it Starfleet, or is it the transportees?
Starfleet is the party capable of overriding and then boosting matter gain and signal. Does this mean Enterprise
was initially doing the matter gain and signal thing, and Starfleet overrode that? Did Starfleet ever manage to override anything?
"Matter gain" is unique technobabble never encountered again, but "signal" is something that in later Trek is part of the overall process and not necessarily pointing to any particular sending or receiving end.
It's possible, then, that
did all the beaming, and the hiccuping machinery made a mess of it
did the beaming until Starfleet overrode
and Starfleet did a joint beaming (although those are a rare thing elsewhere in Trek, and considered a special boosting technique for demanding conditions in TNG "Realm of Fear")
4) Starfleet did all the beaming (although if the Enterprise
platform was merely an inert target, then it's difficult to see how Enterprise
machinery problems would affect the beaming.)
Even #4 is a distinct possibility, as it may be there was nothing wrong with the transporters at either end. Rather, the problem aboard the starship may have resulted in shields being half-raised, or some dangerous source of radiation being activated, with the end result that even fully functional transporters would kill their customers, unless signal were boosted.