Forge of God
by Greg Bear, I presume. The sequel, Anvil of Stars
, is even nastier, especially when it's revealed what the von Neumann machine culture uses as home base... Automated ships appear in many roles in the sequel.
That Kirk and Spock would assume that the DDM is an accidentally released ultimate vengeance weapon, rather than a normally working dedicated berserker, is a bit surprising: the concept of preemptive planet-killers (whether self-replicating or not) had been around for a while already when the episode was written.
We don't have to assume that our heroes got it right, of course. It's pretty enjoyable to gradually find out how wrong our TNG heroes originally were about the Borg! And quite a relief for the UFP that our TOS heroes never got the chance to find out more about the DDM.