True, except Pike wasn't the only one affected by what happened.
BOYCE: Sometimes a man will tell his bartender things he'll never tell his doctor. What's been on your mind, Chris, the fight on Rigel seven?
PIKE: Shouldn't it be? My only yeoman and two others dead, seven injured.
BOYCE: Was there anything you personally could have done to prevent it?
PIKE: Oh, I should have smelled trouble when I saw the swords and the armour. Instead of that, I let myself get trapped in that deserted fortress and attacked by one of their warriors.
So three crew pwersons were killed, seven were wounded, Pike apparently survived without a noticeable wound, making a total of eleven crew members present, and there could have been other crew persons present who were not casualties and thus not mentioned by Pike.
One warrior with sword & spear, etc. vs at least eleven Starfleet crew embers with laser pistols?
Possibly Pike's crew didn't carry any weapons, which could be one decision that Pike blames himself for.
Possibly Pike was separated from the others in the landing party and attacked by one warrior, while the others were attacked by several warriors. That would make the Talosian illusion more similar to the actual events.
Or maybe the Kaylar or Kalar attacked at least eleven starfleet members, killing three and wounding seven.
In the illusion in "Menagerie Part 2":
VINA: The Kaylar!
PIKE: It's starting just as it happened two weeks ago, except for you.
This implies that Pike was alone when attacked by the Kalar, except possibly for a native Rigelian woman. So possibly the other Starfleet people were ambushed someplace out of sight and sound by other Kalars, or maybe this Kalar had already attacked and driven off the other other Starfleet persons and was now about to finish the job with Pike.
VINA: Quick. If you attack while it's not looking.
PIKE: But it's only a dream.
VINA: You have to kill him as you did here before.
This suggests that Pike killed the Kalar in a courtyard of a deserted fortress similarly to the way he killed him in the illusion, though possibly using different weapons than in the illusion.
But it seems uncertain whether there were other crew members present in the fortress courtyard when Pike fought the Kalar, including wounded Spock and Tyler and five other, probably incapacitated, crew members, plus three other crew members lying dead on the floor. That is one possibility. The other possibilitys that Pike was alone in the courtyard, except possibly for a native woman, and released the other surviving crew members from a dungeon or something after killing the Kalar.
So it is interesting to speculate about what hppened on Rigel VII.
As I remember, James Blish's novelization described the political situation briefly, with advanced travelers from other Rigelian planets trying to take over the less advanced Rigel VII, but I don't know if that was in any of the scripts that Blish used.