As much as anything, this stems from plot convenience. When William Hartnell decided to step down, the decision was made that the character would "die" and "regenerate" into a new Doctor, in order to be able to explain away changing the actors. The actual regeneration process changed between seasons and producers and showrunners etc
Yup. At first, it was "renewal," and said to be something the TARDIS did for the Doctor. The second time, it was just called a change in appearance, and it was just something the Time Lords did so the Doctor wouldn't be recognized by his enemies on Earth (allegedly, though that didn't stick). It wasn't until the third time that it was called regeneration and established as a natural healing process. And it wasn't until years later that they established a limit of 12 regenerations (13 lives), retroactively making the Second Doctor's forced regeneration into a sort of partial death sentence, what "The Power of the Doctor" recently called the ultimate punishment, rather than the fairly casual makeover it was assumed to be in "The War Games."
Then you got the timey-wimey weirdness with the Watcher in the fourth regeneration, the first time the Doctor saw it coming. The next couple were pretty straightforward, but then you got the TV movie and Seven's regeneration into Eight being more of a horror-movie kind of thing with lightning. And then you get to the new series with the standing-up regenerations and the huge bursts of golden energy (although that can be seen as an interpretation of the whiteout effect of the first regeneration or the swirly video effects around the fourth, fifth, and sixth).
After "Journey's End," I decided the change in appearance is a side-effect, and not the main process. Maybe once the accelerated healing starts it wants to keep going and going until it's totally reshaped the entire body (and then, similarly, takes a while to wind back down, so the Time Lord is much more durable for the first several hours in the new body), unless the energy is expended in another way.
I think I see it the other way around. I figure the norm is for the regeneration to cause an appearance change -- since that's kind of the whole idea, that the body rebuilds itself and starts over -- but Ten was so intensely attached to his current persona that he somehow managed to override the process and ensure that his new body was a match for his previous one. True, that's not how he described it at the time, but it's supported by the later revelation on Trenzalore that he actually did use up a full regeneration that time, so technically he had two consecutive incarnations that looked and acted the same. And that's further reinforced by the fact that the Doctor has now managed to regain the David Tennant appearance for yet a third incarnation.
There is evidence that Time Lords can sometimes control regeneration enough to choose their appearance -- Romana in "Destiny of the Daleks" being the first example, but we also saw that Twelve's appearance was subconsciously selected to match Peter Capaldi's character in "The Fires of Pompeii" to send the Doctor a message. I figure that normally, the regeneration is uncontrolled and what you get is potluck, but in some cases, a regenerating Time Lord can consciously or subconsciously influence the outcome, either to choose a specific appearance or to keep one's current one.