The thing is, other than these books (Vanguard, Errand series and Cloak), I'm not that familiar with "darker"/"sophisticated" TOS novels, and would love some more recommendations..
The Final Reflection
The Wounded Sky
I guess are fairly sophisticated, although that's not the easiest term to quantify. They're superb reads either way.
I guess traitor winds is a bit darker then normal while remaining lots of fun to read. Windows on a Lost World is darker, but personally speaking I found it really bad. Shadow Lord is darker, but....well it's a fun book, but I can only assume the author only had the most fleeting understanding of Star Trek, as there are rather glaring problems with the entire idea of the story, not to mention other problems.
was writing this out before your second post. given what you said, probably only The Final Reflection meets that - and bear in mind that it is not consistant with canon anymore.