The concept of the colony on Nimbus 3 (Star Trek 5) was sort of similar, however not so well executed.
Where B5 is so different is that in B5, humans are the newbies, relatively. In trek, all the races have known each other for ages.
However as a thought-experiment, how could it have been done hypothetically?
1) Set in the Beta Quadrant, using the Federation, Klingons and Romulans as the main races. Add in a race that has recently gained independance from either the Klingons or Romulans and have the Metrons, Organians or a rogue Dowd or a Q (not necessarily 'The Q') as the higher races pulling strings. This could have also allowed for bringing in other TOS favourites such as the Tholians or the Gorn.
2) Use the existing DS9 neighbourhood, with Bajor having been granted independence with a number of colony systems. The problems would have been getting the 'higher races' in, and keeping the Klingons and Romulans out.
3) Have dispensed with the concept of DS9 as a followon from TNG (yes I know there is an overlap), and done it as a birth of the Federation story set around the Romulan War, where it could be more primitive and darker.
