A pair of more or less obvious possibilities:
1) It was just a ruse. There really were but Romulan peaceniks aboard those ships, no troops at all, and the Romulans took secret delight in having Starfleet essentially execute the dissidents for them - while the many enemies of Starfleet saw a truly audacious Romulan maneuver, one that raised the Romulan stocks a lot among the fellowship of UFP foes.
2) The troops aboard the old Vulcan ships were irrelevant to the whole. The idea was to run a trio of ancient, noisy rust buckets under a flag of truce all the way to Vulcan, this noise masking the presence of a fleet of cloaked Warbirds. We know there was at least one Warbird shadowing the Vulcan ships; with but a few more, the force would certainly have had enough troops and firepower to take the planet hostage.
It does seem likely that what happened was what the Romulans wanted to happen. Data, Picard and Spock escaped way too easily, and Spock was painstakingly provided with a means to transmit a warning message to Vulcan; it really smells of a setup.
It would make for a classic Romulan plot if the idea was to make our heroes send a message of Romulan duplicity when in fact a real peace mission was underway. Or, if the second option is taken, to send a message of Romulan duplicity that would hide the real
Romulan duplicity. At the conclusion of the episode, the Romulans do seem to emerge as winners: they have discredited and uprooted their dissident movement, they have put Federation defenses to shame and frightened everybody, and they have lost no ships of their own while audaciously stealing and then destroying Federation property with impunity.
At no point in TNG does it appear that the Romulans would be interested in open war. Thus, a plotline that suggests such a thing is rather likely to be just another Romulan smokescreen...