I'll take a stab at it, although probably get it completely wrong. What we must remember though, is that the reason that Vanguard
was so different to what went before was the fact that it did take us into dark and uncomfortable places, in contrast to what the crew of the Enterprise
traditionally turned up.
Web of the Romulans
by M.S. Murdock could be a good choice. It's been a while since I've read it, but contains a fair bit of Romulan political intrigue as well as dealing with Starfleet preparing for a war with the Romulans. Again, might not be what you're after.
You're probably familiar with the Rihannsu
series of novels. I've only read the first two so far, but we have treachery, deceit, secret missions, delving into the Romulans yet again.
I've not read it yet, but the description to Dwellers in the Crucible
could be quite a dark novel and offer a dark and twisted view of how the Federation keeps the peace. Have to check it out myself.
by Diane Carey sees a renegade Starfleet admiral and Kirk and crews attempt to stop him and his prototype dreadnought.
The Pandora Principle
by Carolyn Clewes. Good one this, bit of espionage against the Romulans after the Enterprise
discovers a Bird-of-Prey adrift. Also examines Saavik's harsh beginnings on Hellguard. Dark stuff.
I think that's it, by-and-large, apart from what you've already read. Coule be bad suggestions, but I'd read them all anyway because they're good.