Excerpt from William Shatner's...Frontier pg. 62-63
Soon after, one [re: script structural inconsistencies] became apparent. The scene where Spock refuses to shoot Sybok cried out for a deeper motivation that would explain Spock's behavior. At the time, the only explanation given was that Spock knew this Vulcan in seminary school. It seemed an inadequate reason why loyal Spock would refrain from defending his ship and his Captain. Harve, David [Loughery], and my father began searching for the background story which would make the scene realistic.
"We originally said that this Vulcan would be someone that Spock knew," Harve explained. "But at one point I said, 'Ah, it has to be more than that-- it's his brother.' 'Get out of here!' they both said. 'Why his brother?' I'll tell you why I said it and why I fought for it: because none of us could solve the problem of why Spock wouldn't kill him. The answer is that there had to be something else operating there. Just because he was another Vulcan wasn't sufficient. Spock would have found a way. He would have used a phaser, or the neck pinch-- something. So I thought there was only one way. He has to hesitate because he has an emotional attachment. And for him to have an emotional attachment, you have to put a label right on his forehead that says, 'instant understanding' hence, brother."