I have two jobs. One is being a researcher (finishing my PhD in a couple of months), the other is a software engineer for a state/government organization.
There have been ups and down with both jobs, especially the researcher one. When you are doing research, sometimes you feel like you are really doing something special and the world feels great. Other times you feel that no one really cares about what you are doing so it is all in vein. Besides that, I've also had many problems throughout the years due to colleagues that were difficult to be around and work with. But, in the end of the day, I enjoy that part of my life.
My software engineer job also has ups and downs. The main reason being that people in charge are not always aware about the process of writing code so they cannot understand what tasks are reasonable and feasible. Fortunately, my immediate supervisors and managers are nice guys who know how such things work.
I will agree with you that colleagues can make all the difference in the world. Even the best of jobs can make a person feel miserable if they are surrounded by the wrong people all the time. And a less-exciting job may feel great if you spend your day with good people.