Having mainly a mathematics background, my favorite physics books tend towards the more math intensive. My top three books are:
- Techniques of Differential Topology in Relativity by Roger Penrose
- Gravitation by Charles Misner, Kip Thorne and John Wheeler
- Large Scale Structure of Space-Time by Stephen Hawking and GFR Ellis
I also like both Gravitational Curvature
and The Geometry of Physics
by Theodore Frankel, but I'm most likely a little bias towards them as he was a professor in the math department at my school.