Do we know what the relative Rebel fleet sizes are between RO and ROTJ?
Well, the Scarif fleet wasn't
that big. You can easily see the entire group in the wide shots, whereas at Endor there were always more ships in every direction you could look.
According to Wookiepedia, (because I'm not going to count myself when it's already been done) the
Rebels at Scarif had 1 smallish Mon Cal cruiser, 3 Dornean Gunships, 3 Hammerhead corvettes, 6 Blockade Runners (not sure if that includes Leia's), 4 Nebulan-B frigates, and 10 inexplicable Rebel Transports. Needless to say, the Imperials had two Star Destroyers and the Shield Gate (reinforced by an additional Destroyer and the Death Star). So, that's one large ship, four frigates, and 12 corvette-sized ships. And ten transports, God only knows why (the only thing I've seen suggested is that they're fitted with jammers and sensors and other electronic warfare stuff).
At Endor, the Rebels had ≥6 largeish Mon Cal cruisers, ≥5 Nebulon-Bs, ≥8 Blockade Runners, and ≥3 Dornean Gunships (and, oh yes, ≥16 transports to draw fire). They were able to hold off 1 Super Star Destroyer, about 30 regular Star Destroyers and, briefly, one Death Star. That adds up to 6+ large ships, 5+ frigates, 11+ corvettes, and who cares about the transports.
I'd argue that the much larger Imperial force implies the Rebel fleet was larger than the, "The most of one kind of ship we saw on screen at the same time" figures above, just because the Scarif fleet was only barley outgunning two destroyers, while the Endor Fleet was able to slug it out at what could have been 5-to-1 odds for nearly the same amount of time, and didn't seem to suffer as many losses as they did at Scarif. Of course, cinematically what matters is how big each fleet looked, but like I said, the Endor fleet was so big you could never get far enough away to see all the ships in it, which seems plenty impressive to me.