You're probably right. If I could keep things straight with Doctor Who, Battlestar Galactica, Babylon 5, and even Lost, but not How I Met Your Mother, it probably is a sign I'm not a big fan of the show.
In defense of the show, Rachel Bilson was a pretty important part of the show's continuity. First of all, she's a reputable actress (and not one of the dismissable ex-gfs we'd seen in the past). Second of all, it was a key reveal that the yellow umbrella was hers. Third of all, it was a key reveal that unknown to Ted at that point in his life, the future Mother was the roommate, and all the things in the room were hers, and in fact we see her ankle. That single episode answered YEARS worth of questions, while still keeping the big reveal a secret.
I can totally understand forgetting some minor plot points, but as far as major plot points go, this ranks right up there with finding out who the Final Five are or what's inside the hatch.