Just because that is how large a planetary body has to be to have gravity turn it into a sphere doesn't mean that the Death Star is at a minimum that big. What else would Luke assume it is? He's never left Tatooine as far as we know. He hasn't studied the physics of planetary bodies. He sees a small grey sphere in the distance. It is the obvious conclusion to jump to.
Lovely theory except everyone accepts the comment until Kenobi comes to the dawning realisation that it's a space station. To wit Han responds "It's too big to be a space station."
I think Han is well travelled enough to know what's more usual for moons and less usual for space stations.
ETA: OK fine, let's get pedantic about this!
Yeah, there's a shot of the Devastator approaching the Death Star but the scene cuts off before it gets close enough to really give a sense of scale. (Note the SD is still in the key-light and is not having the DS's shadow cast over it at this distance.)
The only time we really get a sense of scale is when the Falcon approaches.
The second one is the most helpful for scale as it gives a good view of the docking trench and the inset alcove where the smaller docking bays are.
Now to put that in the context of what they were thinking at the time, here is a scale chart drawn up by Joe Johnson during production.
Now, here's that information applied to the second shot.
The red silhouette is Johnson's rough scale, while the blue is a more accurate profile, scaled to the official length of about 1600m, all scaled to the Falcon's official length of about 34m. For clarity I highlighted the geometry of the docking trench, with the hanger bay marked as the centre line.
As you can see, even at the much larger official scale, an ISD can easily clear the outer trench. Now go back up and look at that first image and see how thin that trench looks compared to the overall size of the thing. I think it's safe to say the SD's & Rebel ships in Rogue One looked suitably dwarfed by the Death Star.
For anyone who still think they got the scale wrong, I direct you
here for an in depth technical explanation on the concept of perspective.