To answer the question directly - what happened to them, as opposed to if I want to read more of them - it must be that they weren't as profitable. Invasion showed up the mid-nineties, and then Day Of Honor, which was 5 books, and New Frontier, which started as 4 books, and Captain's Table and Invasion and New Earth which were all 6 books, and then trilogies started happening (like Q Continuum) instead of individual installments... none of that would've happened if it wasn't completely clear that those books sold better. By the late nineties, some years more series books were selling than standalones, and that was before substantive changes were allowed.
Okay, you're talking about several very different things here.
Invasion!,
Day of Honor, and
Captain's Table were, in retrospect, umbrella titles for a group of generally standalone books. What's surprising about
Invasion! is how conservative it is; the four books are
exactly like any other
Star Trek books published in 1996, with the only difference being that all four books involve an alien species we've never seen before.
Day of Honor was an umbrella for Klingon stories.
Captain's Table was an umbrella to link six books that didn't
have to be Captain's Table stories. (Which was, to be blunt, a failing of that particular series; the concept was, in itself, intriguing, and none of the books actually
used it as the peg on which to hang the story. When I pitched for
Constellations, I pitched an explicit
Captain's Table story which absolutely
required the Table and couldn't work any other way.) And
Double Helix should have been, as John Ordover himself said a few years later, published under the individual series titles rather than under the
TNG banner; again, it was an umbrella for six generally disconnected books that had a recurring plot macguffin. With these "umbrellas," you didn't have to pick up
all the books; if you didn't care about
Voyager, you didn't have to buy that book.
The trilogies (and something like
New Earth) are different beasts entirely.
The current preference towards ongoing stories, it seems to me, is a natural outgrowth of that; those same multi-book arcs that so clearly outsold the standalones, only now with ongoing plots instead of miniseries.
John Ordover said in the late 90s that the individual books in a trilogy sold three times what a comparable standalone sold. So, the third book of
Rebels would have sold three times what
The 34th Rule sold, basically. I'm not an accountant, but I can see how attractive trilogies look on the ledger. If trilogies sell like that, de-emphasizing singletons in favor of trilogies makes financial sense. It would've been suicide
not to transition to that.
The trick, of course, is to stay ahead of your market and where your audience is and where they're going to be.