I explain it as later writers misunderstanding what the original writer intended. Like the way Spider-Man's spider-sense, which was originally just a heightened physical awareness of Spidey's environment letting him detect nearby movement, sense unstable surfaces, and the like, has been reinterpreted by later writers as an effectively psychic awareness of danger. Or like all the ways DD's radar sense has been interpreted, as discussed in that blog linked above. (In fact, reading those posts, I was struck by how much Lee's original description of the radar sense had in common with his original description of the spider-sense. Lee seemed to repeat himself a lot between Spidey and DD.)
If I'm not mistaken the "explanation" is that he can feel which light waves are being absorbed by the material and which ones are being reflected and is able to get the color from that. Which, yeah, sure.
Well, he might be able to detect a thermal difference between light and dark colors; a darker surface would absorb more light and thus be warmer. I doubt that would translate to color awareness, though.