It's an interesting question. Though the only real answer can of course be - who the hell knows?
We obviously have no real life space federations or political entities of comparable or larger size to compare to. We don't know of any federal democratic entities larger than the Federation in the Trekverse, or at least I can't remember any. It's hard to speculate without much data.
If we had to speculate, though, there are some things we can base that on:
1) We know the Federation seems to work comparably well at the size at which it exists in the 24th century.
2) Larger entities than the Federation do exist, but they're either something not really comparable, like the Borg (which are in many ways basically a gigantic single organism) or authoritarian empires, like the Dominion.
3) We can try to extrapolate and scale up from existing Earth institutions.
If we look just at the posibility of a central government machine ruling over a huge entity, regardless of how it does that, we know the Dominion rules over a space and population that I assume is several times larger than the Federation (we don't know that for sure, but I'd say such an estimate is reasonable). Of course, as an empire, the Dominion's position is made easier by the fact it doesn't have to care about the will or the well-being of the population. But then, it's also probably made harder by the fact that the members of the Dominion mostly aren't really willingly part of the Dominion, meaning there's always a danger of rebellion and resistance. So the empire vs democracy comparison can really go either way. And anyway, we know the Dominion largely leaves it's members to run their own affairs so the practical division of powers between the central government and the lower levels probably isn't that
much different from the Federation.
Now, to take into account the Federation's democratic and federal side, the key institution regarding that being obviously the Federation Council, the Federation's legislature. How large can the Council be before it stops being a functional institution? If we look at real world examples,the Indian parliament has 790 MPs, though they are divided into two chambers. The single chamber European Parliament has 754 MPs. So if we assume the Federation Council is a single chamber parliament with each Federation member state having a single Councillor, we can say there's real world evidence that a Council for at least 700-800 member worlds can still be a workable place. If we take technological advances into account , the number could probably easily be bigger. (The technology argument really works regarding the whole question. Faster travel, better computers, faster flow of information, they all make governing a large polity easier. Also, they foster greater unity.)
If we look at the number of constituent elements, I think the largest number in a federation today would be the 50 states of USA. The Federation already has more members than that. (The UN does have more members but it's not really comparable to the Federation, it's something much, much more looser). How many more can it have, before it becomes too fractured? I don't know, I'd wager at least several times more, though at some point it probably becomes neccessary to form an intermediate level of government. Though that of course opens the question of how many levels are too many?
Can the Federation and it's population become so large that the very idea of a federation loses democratic legitimacy? Well, as long as the Federation's population on the whole has a feeling of willingly being a single unified political body with a common identity, I'd say the Federation can exist as a legitimate entity regardless of it's size. Would achieving such a feeling and identity be impossible at a certain population size, and if so at which size, again, we can't tell. We do know from human history that there's a clear trend from smaller to larger, from the individual tribes of prehistory to today's nations of over a billion people (and even an emerging sense of transnational - the EU - and even planetary unity). And we do know that trend continues in the Trekverse with united planetary governments and then the Federation.
Finally, this all stands only in comparison to what we have today when it comes to governance. It doesn't take into account all the gazzilion possible societal or even biological changes the Federation's population can experience in the Federation's future. Who knows, at some point they might all become energy beings. Or merge with machines. And there's no way of knowing how something like that could be governed.
I'd say a Federation of, say, some 500 members could still work. Maybe even 1000. Any larger than that and it could probably still work, but only with some significant structural changes.