Oh yeah, silver bullets - some of the gun magazines tested this one in the past. The thing is, silver shrinks when it cools (as Kari mentioned). If you cast the bullet in a standard mold, that means you get a bullet that's undersized for the bore of the gun. That means gas escapes around the bullet, the bullet doesn't engage the rifling as deeply, and the bullet may even wobble as it flies. None of these things are good for accuracy.
Come to think of it, sliver is harder than lead and/or copper (yes/no?), in which case engaging the rifling in the barrel may cause more drag on the bullet than normal.