I absolutely love this show. It does start off slow in season 1, but it is totally worth it once you get into seasons 2 and 3. I feel like seasons 1-3 are their own seperate arc from seasons 4 and 5, and I do agree with others that season 4 was weaker. Don't get me wrong, it had some great episodes, but not nearly as good as the build-up that season 3 had.
I will not say much about season 5, only that the set-up that we got at the end of season 4 already seems to be paying off!
I would say give it a shot, and be patient. There is a definite pay-off with this show.