Think of all the things you have missed out on or avoided or didn't get to experience because you said "no." Think of all the adventures and good times and wonderful experiences you could have had, and will have all because you said "Yes."
Saying no to an experience is simply another way of saying yes to a different experience. But I do get what you're saying. I'm definitely the cautious type and I should try more things without putting everything through an extensive audit first.
As for my advice, chill. Just slow down and relax ...