Certainly something like Trek IV was a family film, and was the highest grossing TOS film, right?
In raw numbers, yes, if you define "TOS" to exclude the Abrams interpretation. But corrected for inflation, TMP was actually the highest-grossing Trek film until the 2009 film broke all previous Trek box-office records. Although TMP came out at a time when movies stayed in theaters far longer, so if you correct for that, TVH might come out ahead. But TMP would still have a stronger showing than most of its successors. Maybe it had more audience appeal than you think.
Again, I think it's important to remember that in 1979, the G rating did not have the "kids' movie" connotation that it has today. It was used for films that were suitable for family viewing -- movies that didn't have anything parents might not want their kids to see, like sex scenes or gore, but were not necessarily aimed at children. That's the case with TMP. It was meant to be a sophisticated science fiction film with adult appeal; it was simply kept free of content that might make adults think they had to leave the kids with a sitter when they went to see it.