It wouldn't bother me. We didn't have a color set anyway until I was around seven or eight; I know that probably makes me sound old (I am 40).
Even when we did have color, I watched a lot of shows that were black and white; I Love Lucy, The Dick Van Dyke Show, Beverly Hillbillies....my mind must add color on its own or something since I honestly don't remember them or describe them in black and white.
However, like others are saying, lack of color in film is used in artistic meaning. The more the visuals and mediums 'make a statement', the more work is put into camera angles and the like...I don't know that current actors would be willing to put in that much effort to pull a regular series off. They are rather lazy in their trade, often letting CGI artists do the acting for them.
Actors of that quality: Lucille Ball, Audrey Hepburn, Buster Keaton, Lon Chaney Jr....just to name a handful, are few and far between nowadays, in my opinion.