Played through Pokémon Yellow this past week, it was easier than I remembered. I swear last time I played it after beating the Elite Four & Rival there was a place you could go to fight Ash. But this time there wasn't and I can't find anything on the internet to suggest that's the case, so now I think I might have dreamed it
You didn't dream it, you're just remembering Red in the wrong game. He appeared in Silver, Gold, and Crystal.