I don't believe I can give anything close to a complete definition of what Star Trek is, either, but there are certain elements that I would point to as being at least part of the core of it.
1 Obviously it's a space show with a high technology level. Although that doesn't preclude the occassional story foray into less technologically advanced places where the spaceships can't come to the rescue.
2 It has significant ties to the optimistic ideals of the Federation - reaching a better world through cooperation, tolerance and scientific advancement. These ideals don't have to be unquestioned or even always dominant in all the characters. But they do have to be clearly present and enduring in some form. This is the biggest reason I would say BSG is clearly not Star Trek.
3 It tells a mixture of adventurous stories focusing mainly on the characters, their exploits and their development, often posing challenging moral questions, and (somewhat) less straightforward sci-fi stories focusing more on the possibilities of technology and alien life and the inexplicable wonders of the natural universe, often posing fascinating questions of reality and philosophy.
4 It tends to feature characters that are usually larger than life, but don't really feel larger than life - and are mainly intended to be more or less regular people in what is simply a rather fantastic world.