He was 32 during Buffy's first season, 36 when he departed as showrunner and 38 when it wrapped. Buffy (during Seasons 1-5) was the Mutant Enemy series he spent the most time on set of. He spent a lot of time on the set of Tim Minear's Firefly too (Whedon showran the pilot and worked on the rest of the series as Minear's co-exec/second-in-command), but production for the 13 episodes proper only lasted about four months.
He spent most of his time on David Greenwalt and Jeffrey Bell's Angel in the writers room. He wasn't involved in Season 1 beyond the pilot and writing Buffy's dialogue at Tim Minear's request in an episode. He spent Season 2 showrunning his final season of Buffy (Season 5). He spent a nice chunk of Season 3 working on the Buffy musical during his post-Buffy "vacation" and spent the rest of the season working on the feature-length Firefly pilot. He spent the first half of Season 4 working on Firefly as Minear's co-exec. And he spent a nice chuck of Season 5 working on writing/shopping around Serenity, with filming commencing on June 3, 2004 (after script delays postponed it from the original October 2003 start date).
Sarah Michelle Gellar during Whedon's Buffy run: 19-23 (Noxon's run: 24-25)
Alyson Hannigan during Whedon's Buffy run: 22-26 (Noxon's run: 27-28)
Eliza Dushku during Season 3: 18-19
Charisma Carpenter during Seasons 1-3: 26-28
Emma Caulfield during Seasons 4-5: 26-27
Three of those five actresses have seemed reluctant to work with Whedon again. And Dushku really needed the Dollhouse gig after getting stuck being typecast in crappy horror movies for several years.