I'm not sure I want another story where the Doctor is aware that his regeneration is coming up. ... But we just had Tennant's run end with a story -- heck, an entire arc -- where he had advance warning that his demise was coming, and Smith's whole run so far has been dominated by prophecies about the Doctor's imminent death, and I'm just getting a little tired of it.
I suspect it's the nature of arc-based television storytelling that's driving this.
Eccleston avoided the long-term build-up to his regeneration, but that was due to the uncertainty from the producers as to 1) whether or not the series was going to continue and 2) whether or not Eccleston was going to continue in the role if the series did continue.
Since then, however, you're right in noting that both Tennant and Smith have their inevitable ending overhanging the stories themselves. And I think that the example of series like Lost
is the reason because audiences are primed now to expect apocalyptic changes at the end of seasons. But those apocalyptic events can't happen in a vacuum; instead, they have to be hinted at, built towards, and made unavoidable as the episodes tick down toward the inevitable conclusion. So long as Doctor Who
is made in the style of arc-based genre television, I doubt we'll see another shocking, non-apocalyptic regeneration like "Caves of Androzani."
In the particular instance of Matt Smith's regeneration to Peter Capaldi, I'd like to think right now that the reason Smith's Doctor was so freaked out by the appearance of John Hurt is that he already knows what happens after that point because he's already witnessed his regeneration through Hurt's eyes so he knows that when he meets Hurt's Doctor that his end is nigh. (I know that it's been said that Smith will be in the Christmas episode. I'm not convinced by that yet.)