A better question is why he was living in a cave when he knew a Starfleet base was within reasonable walking distance.
You seem to be misunderstanding the timescale here. The entire sordid affair took mere hours, from Nero's capture of 24th century Spock to Nero's defeat in the hands of 23rd century Spock. Within those hours, Nero dropped the older Spock onto Delta Vega - but obviously making damn sure Spock would need more hours than Vulcan had left before he could reach the local Starfleet installation. Otherwise, Spock could warn Vulcan.
Spock would thus indeed be on his way to the installation, but would be too late already (again!). So it would be time to rest - in a cave if one were available. Spock could later complete Nero's sadistic scenario and reach the installation, thereby surviving to grieve.
(Although we don't really see Spock resting in a cave. For all we know, he was walking outside, carrying fire-making equipment provided by Nero lest the Vulcan die of exposure, and saw Kirk and the beast stumbling into the cave; he then followed them in and evicted the beast. After that, he would set up camp there.)
Timo Saloniemi