I'm not sure about ROTJ, but I'd assume that it would have been a combination of different scale models and matte paintings. As far as scaling a model goes, sure, they could do that, but it's still more challenging to get a large model, shot at the correct angles into the frame. Plus, you have to remember that motion control means that you need room for the model to move. You could get better, more intricate movement with a smaller model.
In TMP, there was an extremely tiny refit-E model that was used for the V'Ger flyover. Less than a foot, I believe.