Honestly, I think the dialogue is suggesting that the trauma which blew the other crystals put a crack in the 4th one (suggesting 4 crystal assemblies). But your explanation is also quite valid - maybe that's why it is the spare
The Alternative Factor
features 4 Dilithium "paddles" which (to my mind) echo the 4 crystals in Mudd's Women
Finally, Kirk negotiates for 6 crystals with the miners. Either the Enterprise has the capacity to run on 6 crystals, or he was thinking ahead to the next incident, and wanted spares!