I would argue that the size of the Federation is pretty much maxed out given current communication and travel speeds.
A larger realm would need faster ships or better commincation technology like the comm array the Hirogen were using.
How does represenative government work if it takes years for your ambassador to transit to Earth?
We need Voth transwarp, Cytherian drive, slipstream, Iconinan tech or other for a truly expansive Federation.
USS Voyager already brought back ample technology to allow the Federation to develop Transwarp and more likely Slipstream propulsion. With Slipstream the entire Galaxy would be opened up to the Feds. It seems reasonable to assume that the Federation wont be limited to simple warp when other species in the Galaxy have Transwarp capability. Maybe the Federation don't develop it themselves but instead obtain a new member world of a species that has already developed it and is willing to share.
Communication wise It stands to reason that with faster propulsion comes faster communication.
I can't see some species never wanting to join the Federation.
The Romulans, Cardassians, Klingons, Ferengi, the Founders to name a few. All are antithetical to Federation values, since Federation values are based on the human Enlightenment and liberal ideals. What liberalism do the Romulans, Cardassians, Klingons or the Founders believe in? lol. It's also a matter of the nature of other species. Romulans by nature are manipulative, calculated and deceiptful. Cardassians by nature are militaristic and cruel. Klingons by nature are aggressive and warlike. And the Founders by nature believe in order and rigidity. The Ferengi would most likely see Federation values as undermining their way of life.
I think for the most part, species that join the Federation are most like it in terms of outlook.
By the end of DS9 the Cardassians were a defeated people, much more open to the Federation and the Federation were helping them get back on their feet. Even Damar before his death became a new man and even before Cardassias joining of the Federation the Cardassians were more open to peace talks with the Federation and more notably the Bajorans. Don't forget that the military dictatorship that ruled Cardassia was overthrown and the Detapo council (rulers of the people) took power.
The Ferengi if you don't recall ended up with Grand Nagus Rom and his son Nog is a Starfleet officer. Rom was going to change the Ferengi civilisation and make it more tolerable of womens rights and what not.
The Klingons have been steadily working their way to becoming Fed members every series we saw. They became Federation allies, fought by the Federations side in the Dominion war and best of all it's now led by Martok who's spent a lot of time with Starfleet and starfleet officers. His best friend and house member is Worf who served years in Starfleet, grew up on Earth and is now Federation ambassador to Quo'nos.
Romulus is destroyed, what was once a tightly controlled empire is now an Empire in turmoil looking for some kind of help and a new direction. If the Federation does well in aiding the Romulans there's every chance they will come to respect the Federation. Before Nemesis the Romulan senate was more open with the Alpha Quadrant, even the newly elected senate denied the use of Shinzon.
The main problem with the Cardassians and Romulans were the Obsidian order and Tal Shiar. Once the Obsidian order was gone we saw the Cardassians rising up and taking power and kicking out the military dictatorship. With Romulus destroyed and the Tal Shiar likely weakened or also obliterated the Romulans are more free to make their own decisions.
As for the Dominion, Odo has every opportunity to open the Founders up to being more tolerant of solids. He's got years of memories and emotions to share with the link including love for a solid. In time even if the Dominion does not join the Federation or disband itself and allow it's subjugated worlds to join the Federation it can at least have a peace accord with them. More likely though IMO the Founders may eventually agree to disband the Dominion in favour of the Federation.