Does he though? We only see him tell Tom that he can see into the future. The other characters fluctuate between referring to him traveling through time or simply seeing the future, but we never find out what details Sam gave up. (Although his sister does refer to "swiss cheese memory" so I grant you it's quite possible that he gave them a fairly full account.)
I'm not saying it can't have happened, I'll just revert to my original sentence, "Personally, I wouldn't consider Tom calling Magic 'Little Brother' as proof that he somehow knew that Magic was actually Sam." Evidence maybe, but definitely not proof.