There's a fundamental disconnect between LOTR and Star Trek. Tolkien was trying to preserve an idealized past while Roddenberry was jumping ahead to an idealistic future.
Sure, you can switch LOTR around so that the goal is to reach the future rather than preserve the past, but then you've gutted LOTR and replaced it with Star Trek rather than fused the two.
If you ignore what LOTR is actually about and just preserve the trappings, then I'm sure they could make a good Star Trek story. Sauron is some evil, superpowered megalomanic alien, etc.