Actually, I believe Singer voluntarily left X-Men 3.
Warner Bros. offered him a sweeter deal with Superman
, and so he accepted. However, you could say that 20th Century Fox was dicking him around by prolonging the negotiation process for X-Men 3
so it was their pis-spoor management that led to Singer's departure. Singer was more than willing to direct X3
had Fox waited for him.