There's an explanation deriving from the Voyager episode "Deadlock". In that episode, a spatial anomaly splits the entire ship in two, but the two Voyagers are not *quite* in the same reality. Imagine if you will, that they were in two parallel quantum realities which split at that moment. Reality splitting happens all the time, it would seem. The odd thing there was something prevented the split from completing normally.
So let's say the same thing happened with Riker. It was a dangerous beaming; in one quantum reality it succeeded, and in another he was reflected back to the planet. In all likelihood he was disintegrated in a third, but that doesn't matter here. What if the anomaly wasn't a transporter split at all, but simply a transit between quantum realities by the reflected transporter beam so it ended up in the wrong one?