Actually, I believe Singer voluntarily left
X-Men 3. Warner Bros. offered him a sweeter deal with
Superman, and so he accepted. However, you could say that 20th Century Fox was dicking him around by prolonging the negotiation process for
X-Men 3 so it was their pis-spoor management that led to Singer's departure. Singer was more than willing to direct
X3 had Fox waited for him.