Having Judi Dench serve as "M" in both universes was more of an aesthetic choice made by the filmmakers, Dench being as well-regarded in the role as she was -- it was simply to have one familiar, returning touchstone available for audiences to latch onto, while rebooting everything else (very similar, in many respects, to Leonard Nimoy).
Right. The filmmakers don't expect anyone to think these movies represent a realistic, consistent universe, particularly not when they've been recasting Bond, Leiter, and Blofeld time and time again for decades and ignoring the passage of time. Having the same actress play the same role in two continuities is no worse than having different actors play the same role in a single continuity.
Not to mention that it's happened before in Bond films.
Never Say Never Again is in a separate continuity from the mainstream Bond films, in that it features a Bond who still looks like Sean Connery operating in the late '70s, a time when the mainstream Bond looked like Roger Moore. So if Connery could cross continuities, why couldn't Dench?
And there are certainly other examples of different continuities using the same actor in the same role. Adam West has played Batman in at least three different realities, the '66 live-action series, the '77 Filmation cartoon, and the early-'80s
Super Powers Team iteration of Hanna-Barbera's
Super Friends. Similarly, Kevin Conroy has played Batman in the DC Animated Universe, four different DVD-movie universes (including the animated version of the Nolanverse), at least three different video-game universes, and a comedy short on DC Nation. Michael Clark Duncan played the Kingpin in both the
Daredevil movie and the MTV CG-animated
Spider-Man series (which was supposed to be in continuity with the first Raimi movie but was contradicted by its sequels). And J.K. Simmons has played J. Jonah Jameson in the Raimi Spidey universe, the
Avengers: Earth's Mightiest Heroes universe, the current Marvel/DisneyXD animated universe, and the LEGO Marvel universe. And Peter Cullen has played Optimus Prime in several different
Transformers continuities; ditto Frank Welker as Megatron.
Admittedly, there are fewer examples of it happening in live action. But it can happen. Maybe the
Highlander franchise qualifies; the various films and TV series seem to represent several distinct variant continuities, but Christopher Lambert has played the same character in most of them.