I guess we'll have to assume that the issue was reintegrating them out of the transporter pattern buffer - and that in the case of TMP, uploading them into the buffer didn't happen correctly in the first place.
Let's also consider the circumstances of the two different transport attempts. The "Obsession" transport sequence is begun seconds before an anti-matter explosion, and Kirk and Garrovick apparently have made it as far as the pattern buffer when it hits.
In TMP, there's no anti-matter explosion, but the failure is due to defective equipment.
VOICES (OC): Check. Cleary on number six.
Space matrix restoration coils. Dilithium crystals.
I knew it. The transporter sensor was not activated.
SCOTT (OC): Cleary! Put a new backup sensor into the unit.
CLEARY (OC): Aye sir.
[Kirk announces he's taking command]
CLEARY: Transporter room, come in! Urgent! Redline on the transporters, Mister Scott!
SCOTT: Transporter room, do not engage! Do not...
CLEARY: It's too late. They're beaming now!
[Enterprise transporter room]
RAND: Do you read me Starfleet? Override us. Pull them back!
KIRK: Give it to me. Starfleet, boost your matter gain, we need more signal! ...More signal!
SCOTT: We're losing their pattern.
Along with Kirk telling Starfleet to boost the matter gain while a defective transporter sensor is in the process of being replaced. No telling what that did to the circuits.