I don't know about that. I've written a story where it's impossible not to be conscious that you are reading a story, and that was a deliberate choice on my part. Some stories function best artistically if they keep their audience off-balance and at some distance.The last thing an author should be doing is reminding the reader that he's reading a novel.
Assuming you're talking about "Make-Believe"... once it's clear what's going on in your story, it doesn't distance the reader. The story wouldn't have the effect it does otherwise. It just so happens that we're then drawn into a different kind of story than we originally expected.
As for Trent's remark about Immortal Coil: I don't see that as the same phenomenon. The references in that book were in-universe and appropriate, if arguably underexplained for the benefit of people who haven't seen TOS. The main out-of-universe reference was one that I would guess escaped more than 90% of the book's readers (Vaslovik from The Questor Tapes).