Force the orifice open with a tractor beam? Not likely.
Scans couldn't penetrate it, and the whole reason Spock went on the spacewalk in the first place was to get information about the inner chamber. I would assume that communications were blocked too. In that case, there would be no way to get a lock on Spock to beam him out. I doubt the transporters could penetrate either, for that matter.
Phasers and photons, maybe, to blast a hole in? Sure, if Kirk would like to risk getting the Enterprise digitized, assuming they're even effective at all.
Yes, Spock would probably stay in beaming range, for whatever that would be worth. It would be a case of "so close and yet so far".
The following excerpt below is from memory-alpha, here
is the link to it.
A tractor beam was an attenuated linear graviton beam used by starships and space stations to control the movement of external objects. The tractor beam placed spatial stresses on the object in specific areas, allowing it to hold the "tractored" object in a fixed location or alter its position and/or trajectory. While tractor beams were normally used to pull objects towards the beam source – usually to tow objects – conversely, they could also be set to repel objects. They were most often employed to tow a disabled or vulnerable ship to safety, or sometimes, used by an enemy ship to control the movement of the target vessel.
So I could easily see 4 shuttlecraft using their tractor beams, along with their engines, to exert enough force to either pull or push the orifice open wide enough in the 4 directions (up, down, left, and right) to allow the 5th shuttlecraft through it to rescue Spock.
I doubt the scanners operate at the same frequency and amplitude as the transporter beam. We have seen people transport through solid rock in the pilot episode "The Cage" and the Season 3 episode "The Cloud Minders". I would think the transporter beam's frequency and amplitude would be some where between x-rays and gamma rays frequency and amplitude in order to penetrate solid rock.
However, for the sake of argument lets say you are right and the transporter beam cannot penetrate the walls between V'ger's chambers. I still think rescuing Spock by beaming him back would be possible if the Enterprise launched a group of probes to act as transporter relays. For example, say each probe is about 1 yard (or 1 meter) or less in diameter and each probe is 25,000 miles away from each other at maximum. Each node in the network relay would be in contact with the node 25,000 miles in front of it and with the node 25,000 miles behind it, again that would be at maximum distance, though I doubt in this case the probes needed to be that far away from each other to locate Spock. With the the first node at the orifice with direct line of sight with the Enterprise's transporter beam and this first node is also in contact with the next node up to 25,000 miles away from it. That first node would be partially in both chambers to connect the Enterprise with the other relay nodes. Thus, Spock is beamed aboard the Enterprise.
Navigator NCC-2120 USS Entente
Stokely: In an insane society, the sane man must appear insane.
Harvey Holroyd: Where'd you get that?
Stokely: Star Trek. [leaves the room]
Harvey Holroyd: [to himself] God, I miss that show. (Source: "Serial", 1980.