Other than the word "matter" in the name, they're nothing alike. Trek's also dabbled with Dark Matter (a real... well, hypothetical real thing) in Voyager.
According to the novelverse (DTI: Watching the Clock
), Red Matter was discovered in 2383, and at that time had only been synthesized in microscopic quantities. And they didn't actually know how a runaway chain reaction of any significant quantity would would end.