I just started listening to Judge John Hodgman, which is excellent, and at roughly 20 minutes, each, perfect for my commute.
While I don't find the hosts annoying, I do listen to the Nerdist only when I like the guest. Their chats with the celebrities really touch upon the guest's history and if I don't know the guest that well, then I don't get much out of listening to them. I tried listening to a few "hostful" podcasts without guests, but they weren't enough to keep me interesting. That being said, Interviews with people like Penn Jillette, Adam Savage, Craig Ferguson, Neil Gaiman, Neil deGrasse Tyson, and John Hodgman have been excellent.
The Nerdist got me interested in Star Talk Radio with Neil deGrasse Tyson. I've listened to a few so far, but man, his early co-host Lynn Koplitz is very, very annoying. I'm glad they ditched her in favor of rotating co-hosts. Sometimes, though, I feel they don't go far enough into the science, since they have pre-recorded interviews, which Neil introduces and chats about with the co-host, who is usually a comedian. Still, it's an interesting podcast and I am learning more about the world around me and the universe, which is cool.
I've listened to NPR's Wait, Wait, Don't Tell Me podcast for a few years now and love it. There have been times when I've cracked up in my car listening to them.
I used to listen to le Show with Harry Shearer, but I never loved it. Now that I've found other podcasts, I leave it until I have nothing else to listen to.
I also listen to the Reduced Shakespeare Company Podcast, but like Nerdist, it depends on the subject/guest.
Every 2 weeks I skip through iFanboy's pick of the week podcast to listen to their thoughts on the comics I bought during my bi-weekly comicbook shop trips. Through them I've listened to Word Balloon and the Aw Yeah podcast. All very entertaining, especially the interview with Mark Waid.
Speaking of Mark Waid, a few years ago he was on KCRW's Guest DJ Project and I started listening to that podcast, again, depending on the guest. Some of the guests have been interesting to listen to, sometimes for their musical choices, but also for a little insight into their past and their creative process.