Well as we all know Season 1 is by far the best because of its simple yet effective premise and endearing characters. There is indeed a noticeable drop in quality in the following Seasons but that doesn't make them bad. I honestly don't understand why the show was cancelled, it still had potential.
I remember hating Season 4 when I first saw it but I watched it again and it's not that bad. It's a very mixed bag and kind of all over the place, it should have been shorter like Season 2 and more focused on Samuel (who is an interesting villain and very well played) so the writers did screw up by having too many things going on at the same time and therefore losing track of the main story. Still I feel the last episode was satisfying.