I think the planet was pre-warp before Osyraa showed up, and they've been isolated from any capital starship help (other than Osyraa's) for a century (since the Burn). Also probably explains lack of replicators on this planet (Osyraa won't give them any).
So it makes sense within the context of what we've been told. What's more troubling is that I'm sure Osyraa is menacing tons of planets. Book's planet gets saved just because he's romantically involved with a Starfleet officer? Why didn't Vance call Burnham and Book out on that? Honestly that conversation was overdue when Burnham disobeyed orders in 'Scavengers' on Book's behalf.
How isolated/backward can they be when Book is galavanting around in his own ship? People from 900 years in the past fixed their problem as an afterthought, and all it took was shining a light on the power of two dudes’ brotherly love.