For us it isn't a matter of "me time", its just a matter of differing interests.
I like to go the pub and watch the game with my old college buddies once in while. Sometimes after that we'll go to a bar. I do not hit on chicks at the bar like I'm single (as
Galactus seems to think anyone who goes to a bar
must), but sometimes I dance, or (if I'm really drunk) do karaoke.

Sometimes a group of us will go for a two or three day hiking/camping trip.
I'm not doing it to get away from my wife, I'm doing it because I enjoy it and she does not.
Sometimes my wife will go to a romantic comedy with her girlfriends, and I will stay home. It's not because she wants to be apart from me, but rather because I'm not interested and she is.
We still spend 95% of our free time together, but we recognize that we are not obligated to accompany one another at all times. Nearly all of our friends are mutual. If I have a day off and she doesn't, I may go for coffee with one of our female friends. We are comfortable enough in our relationship to realize that this isn't a threat.