Ah, that's right! Too bad they didn't put that in the movie. Saruman would look like less of a chump (or traitor, or just plain in denial) if he voiced that kind of suspicion instead.
Interestingly enough, if it's in The Silmarillon
and not in the Tale of Years, Peter Jackson and the others don't have the rights to film it.