Volume... well, there are two numbers you MIGHT be talking about there... actual volume occupied by solid material, or "displacement" (not really an accurate term in this case, since you're in space, but oh well)... the volume occupied by everything (including open space) inside of the outermost surface.
For "displacement," well... the easiest technique would be to buy or build a model, make sure it's air-tight, and then dunk it in a tank of water and see how much water it displaces. Alternatively, you can approximate this by calculating the volume of individual parts (say, a generally-equivalent cylinder for each nacelle, etc) and add them up.
For the "solid volume," well, that's different. If you use the "model" solution, you'd have to make sure that your model is extremely accurate, inside and out (down to the thickness of deck plating and the cross-section of every beam)... then make sure that there are no "sealed" volumes inside, and dunk it in water to see how much water it displaces. Or, you can do the same thing (with the same requirement of 100% accuracy) in a CAD program and it'll spit out the real volume.
There is no other way to do this. And while you can get a very reasonable and accurate "displacement" value, it's essentially impossible to get a "true solid volume" number. The best you can do is get a fair approximation (which is where I'm going with my version of the 1701) and make an educated guess on the "real" volume from there.
Mass is even harder... because every element in the design has a different material density. In CAD, we do this sort of thing all the time, but that's at a much simpler level... a motor, for instance, may have 100 different elements, each which has a unique volume, and you may have 50 or so different materials, each of which has a unique density. But by assigning the correct material properties to each, and getting the design details right, we can get the software to give us very accurate numbers for what the real part will look like (mass-property-wise, I mean).
For something like a TOS ship, I think the approach I'm taking is the best possible option. Since I'm modeling most of the solid volumes as solid, my approximations for the internal volume and "overall density distribution" should be reasonably accurate (say, within +/- 5%). I have NO IDEA what parts of the ship would be what density, but I"m going to treat the main hull as (for calculation purposes) basic steel, and will assign an additional value to some of the elements in the warp nacelle (to determine how much more dense the nacelles have to be for the impulse thrust vector to be appropriate for the location of the engines).
That's sort of long-term stuff, but I do hope to come up with a reasonable number.
The REAL answer to your question, honestly, is "they make it up." I hope that my approach will be the most "reasonable" answer which has been "made up" so far, though.
