Because as of
"The Gathering" and its like, when a Jedi constructs their lightsaber is when they "give it life" so to speak. Being described as useless otherwise. Going by that aspect, when a lightsaber is
constructed via meditation and the Force, is when the crystal is bonded to the Jedi. Likewise, a Sith would cause a crystal to "bleed" during the same process, but instead of bonding with the crystal, they are dominating it, which appears to cause it to turn red, rather than another color. It is on construction that the crystal and the Jedi or Sith bond via the Force in a deeper way as the lightsaber is built, thus this appears to be when the crystals color changes to reflect that connection. This provides a reasoning for Ezra's blade to be blue one season, but green the next, when it us assumed to be the same crystal in both lightsabers. The base for the discussion of course is related to how Ahsoka got white bladed lightsabers when constructing her newer set.
The bond with a Jedi does not mean the saber cannot be used by others, it just makes the user more in tuned with the saber. However it appears if this is not done, the saber will not work at all. So perhaps there is something involved with the Force to make the crystal work with the lightsaber's components as the Force users builds it, otherwise it just doesn't work as advertised. Building it wrong could even blow up the lightsaber and damage a bunch of stuff around it, yet leave the crystal entirely intact.
The question that has not been answered though is why a lightsaber might be blue for one Jedi, but green for another, or yellow, and especially, why a single Jedi might construct lightsabers with different colors at different times.
Ezra making two lightsaber (likely with the same crystal) with two different blade colors. (though I wonder if Ezra's first blade was what would traditionally come out for someone trying to be a defender, while the green one was because he saw one and went "that looks so cool".)
Ashoka making her first lightsaber green, while her second was a very yellowish-green (the later second set being white was described in the novel)
Jedi General Krell had two double bladed lightsabers, one blue and one green. We can guess he has four crystals.
Count Dooku's dark sider assassins do not apprear to have constructed their own lightsabers, but were given to them by Dooku.