Honestly, I'm not sure there's really a hard line in the sand there, or that the novel continuity was ever "officially" rebooted. If that's the case, I certainly never got that memo.
I still reference vintage Trek books--like "Yesterday's Son" or "Vulcan's Glory"--in my TOS novels when it seems appropriate.
Well, it was never more than a loose continuity in the first place, much vaguer and more intermittent than what we have today. It emerged gradually as authors started cross-referencing each other's books, and there were always books that didn't fit into it. But once TNG came along and Richard Arnold cracked down on having
any continuity between books, there were no more references to the older stuff, and over time, TNG contradicted a great deal of what was in those books. These days, with no more Arnold restrictions, we're free in principle to reference older books, but a lot of them are incompatible with the modern TV continuity.
And it is rather left to individual authors to decide which books still "fit." For instance, I wouldn't reference
Yesterday's Son because it ignores the animated series (it's two years after the end of season 3, yet they've never been back to the Guardian planet in the interval and Bob Wesley is still a commodore rather than a governor). And I always had my doubts about
Vulcan's Glory's portrayal of the Pike era, with so many Vulcans in the crew and Scotty already being aboard and so on. But in cases where there isn't an overt contradiction with canon, it's sort of a judgment call.
And even in cases where books clearly don't fit with modern canon, authors have still referenced ideas and elements from them -- for instance, incorporating Duane's Rihannsu language into modern Romulan stories. It's like DC Comics incorporating Jimmy Olsen from radio and Harley Quinn from TV into the comics continuity -- it's still a separate version of the reality, but it can have specific characters or elements in common.
Granted, there may be more of an effort these days to maintain some sort of inter-book consistency, particularly in the TNG-era novels, but that doesn't mean that everything before a particular date has been cast into the outer darkness.
It's case by case. The oldest book I still count in my personal continuity is Howard Weinstein's
The Covenant of the Crown, because nothing has ever come along to contradict it. But I tend to be pretty picky about what I count in the main continuity -- especially in the TOS era, since the finite amount of room in the 5YM is of concern to me. Sure, there's no need to limit the number of 5YM stories you can write, or enjoy as a reader, but they can't all fit in the same timeline as each other.
Correct me if I'm wrong, but I always assumed that the "Relaunch" label simply referred to stories that took place after the actual shows went off the air, not to some sort of sweeping DC Comics-style reboot.
Well, Marco Palmieri intended it to refer specifically to the initial promotional push for the DS9 post-finale novels. After all, a launch is a single event, the beginning of a voyage, not the ongoing voyage itself. But the term got co-opted to refer to post-finale books in general.
And, yes, the new continuity emerged gradually, more because the authors and editors wanted to coordinate and cross-reference than because of a formally declared plan or policy.