Yeah, the lack of color is the biggest problem with e-paper with refresh rate a close second (though that's been improving pretty fast). But there's some cool tech coming down the pipe, hopefully in the next few years, that will help quite a bit. But for a lot of people, color isn't such a big deal... if all you're doing is reading novels or newspaper articles you don't need it.
As for "one-trick-pony over a multi-talent"... the obvious comeback is jack of all trades, master of none.
If you want a device that can do a little bit of everything, then something like the iPad is great. If you want to do a lot of a single thing and don't care about the other things... why pay extra for something that isn't as good at your preferred task?