BSG actually wasn't successful ratings-wise. it was critically aclaimed, and therefore SyFy stuck with it, despite not getting the ratings they would've like, for the prestige. The current "Lighter" crop of shows SyFy airs now, get better ratings then BSG did, and since TNG still does good ratings in reruns, another series in that light tone on SyFy actually might be quite successful. BSG was definitely very good, but, for me, sometimes it was tough to watch, because it was so weighed down with dreariness, a Trek show, certainly shouldn't be that heavy overall, it should only use darker elements sparingly, when they have something to really say in the story with it.
I actually agree that, yes, a new Trek with darker elements or episodes or arcs, could be very good, but, as stated, it shouldn't be BSG depressing where you want to slit your wrists. It's totally possible to explore darker elements without weighing the show down as a whole with dreariness. DS9 went to some very dark places, but, it only dipped into that darkness, it didn't wallow in it for 7 straight seasons.
But, as Temis says, it all depends on who carries it as to what will work. Most importantly is character development and consistency, don't have characters flip flop back and forth each week at the speed of plot, if a character changes (which they should change and grow), there should be real, plot/arc related reasons, it shouldn't just be random due to "shoving the characters into the story you want to tell this week". You should grow the characters to fit your story and you should write the story grow your characters.