No, it didn't have anything to do with signs of malfunction.
Which is what I meant with my "in retrospect". That is, "taking into account later evidence", that is, reconciling with the fact that what we saw in TMP does not hold true for the rest of the appearances of that ship.
Associating blue with malfunctions or subpar performance is in keeping with the bulk of evidence. And associating TMP with malfunctions or subpar performance basically flows from the premise of that movie. Heck, Scotty thinks he needs four full days to drag that wreck of a ship from Earth to Vulcan!
Actually, the RCS system wouldn't put out any visible flames in space.
Which is why it's weird when it does, and very prominently so, in TNG and DS9 and the new movies and DSC all.
And weirder still that it doesn't do that all the time, but only on certain occasions.
If you watch when Spock's shuttle docks with the Enterprise, you can see the RCS thrusters glow brighter as they are being used to maneuver for docking. I read an interview with Andrew Probert once where he stated this was incorporated into the Refit model but I never saw any on-screen footage that showed it being used.
There were all sorts of blinkies on the models, some working appropriately for their intended role, some most probably not. If the Vulcan shuttle RCS glow was done with blinkies rather than in postproduction, then it's all the likelier IMHO that it would have been omitted from the big ship because of complexity (and fans would in any case still be pointing out they lit up all the wrong thrusters for the observed movement!).
Personally, I find it pretty repulsive how Star Trek technology was downgraded so drastically for TMP, basically becoming 1970s NASA and making it quite implausible that these ships could ever make it to the stars. Airlocks, docking ports, steering rockets... Why not staged propulsion while they are at it, with jettisonable fuel tanks at the very least, and rocket stages falling off the hero ship as she accelerates away from Earth at worst? But invisible RCS even if from visible nozzles is somewhat palatable, and in keeping with how no glow of any sort was previously or later required for making these ships turn in space.
Timo Saloniemi