'Red Dragon' is a prequel to 'Silence of the Lambs' and 'Hannibal', and 'Hannibal Rising' is a (terrible) prequel to all of them, so no, you can watch all of them without seeing Red Dragon and have no problem understanding the narrative. The only major connecting characters between Red Dragon and Silence of the Lambs/Hannibal are Jack Crawford and Lector (and Barney the guard, I guess, but he barely qualifies), and no significant reference is made to the events that happened before Starling meets Lector.
Red Dragon is not as bad as some make it out to be in my opinion. For being a Brett Ratner film it's actually pretty good, and Hopkin's presence elevates the material, though I would recommend the earlier and superior Michael Mann version of the novel Red Dragon, which is called 'Manhunter' and predates even 'Silence of the Lambs.'
Brian Cox plays Lector instead of Anthony Hopkins, and William Peterson from CSI plays FBI agent Will Graham, in essentially the same role (interviewing Lector to gain insights on a serial killer) that Foster's Clarice Starling had in Lambs. Edward Norton plays Graham in Red Dragon, which also adds a scene where Graham wounds and captures Lector after being greviously wounded, which was only allluded to in Manhunter.
I would recommend watching both films eventually, as they both have their strong points, but it's not necessary to do so before watching the three films you mentioned above.