I remember there was a Legends novel about a Jedi knight who went into hiding after Order 66 and eventually, rather coincidentally, reunited with his birth mother, who we found out had regrets and second thoughts about handing her child over to the Jedi. She was then surprised to find out from him that almost all parents voluntarily had their children over to the Jedi because they like the bragging rights of telling others "my baby is going to become a Jedi."I get the feeling that most Jedi came from families who thought it was a better life for them.
We do know in Disney canon, Count Dooku's family found out about his Force abilities while he was still an infant and left him outside their manor with a note attached that said something like "Jedi Property."