I think that Sybok being Spock's brother also works thematically, with Sybok's journey in the film being kind of the flip side of Spock's in TMP. http://www.trekbbs.com/threads/tff-as-thematic-counterpoint-to-tmp.137506/Shatner, Harve Bennett, and David Loughrey said in Star Trek Movie Memories that they made Sybok into Spock's brother because it was the easiest, most efficient way to explain why Spock would betray Kirk to follow him (an idea that was toned down after Nimoy objected, but is still present in Spock's refusal to shoot Sybok). People typically have stronger bonds with their siblings than they do with their cousins.