Ignoring someone doesn't infringe their rights to speak. It just means I choose not to engage them.
Your exact words were "Can we just ignore this guy now?"
That doesn't sound like you choosing not to engage him. That sounds like you telling all of us what to do and belittling his opinion. So let's not kid ourselves as to your perceived intent.