The biggest problem with using established side characters from the show (Dukat, Garak) is the need to explain them. There will always be people going to a Trek movie just for shits and giggles, people who have never seen TNG for example. When you then do a movie about Q, they're going to wonder just who that is, without using half the movie to 'explain' Q to non-Trekkies.
The audience isn't stupid. There's nothing about the characters you mentioned that can't be explained in a very short exposition. Actually, all you need to understand is what they are (Dukat: Cardassian leader, Garak: former spy, now tailor, Q: omnipotent being) and that they have a history with the main characters. You don't even need to know what history. If it's relevant to the story, it will explain itself during the unfolding of the plot. If it's irrelevant anyway, nobody needs to know it. Seriously, it's not that hard, it happens in every single film
And still, there will be people that are put off by it. That's one of the reasons that Trek movies didn't do all that well compared to other movies. Some non-Trekkies go see them, sure. But most people think 'damn, I probably need to see the shows a lot to get the movies'. So yeah, I stand by my statement, using 'villians' from the show that need explaining won't work. Even with a few short scenes, since you can't help but wonder if you would have enjoyed the movie more if you had seen the show.