It's probably the biggest factor. The Klingons felt they would lose it if they joined the Federation. Other planets are shown to not want foreign influence or be drawn in to wars.Self determination is a big factor, too
The Federation also represents an obligation in some way in support of other members and what Wars they might fight. There's a lot of conflicts that the Federation has been involved in. Some worlds might not want to participate in those obligations.