I think it would be possible to reboot
Star Trek in a mild way without adhering slavishly to established continuity. Hell, it's been done in comics for decades upon decades.
If you want to stay in the established continuity then I think it's also possible, but you'd have to pick an era with some breathing room either pre TOS or post TOS such as the TMP era after TMP. Or if you don't want to do TOS then do post TNG, although I don't think there's much demand or sufficient market interest there.
The other alternative is a drastic reboot of some kind. Keep some things like some familiar names and references yet still very much a clean sheet start.
I don't think you can do
Star Trek strictly as it was done with TOS. I think you can maintain many positive elements of TOS that worked, but because certain expectations and tastes are different from then you'll have to update it and do some things differently.
One thing that you have to decide is whether you're doing feature films or television series because that can affect what you want to do and how you can do it.
Firstly, I'd want it to be
Star Trek as it was when TOS and TNG were at their best. That's its identity and not Trek done as
Star Wars or nuBSG or what have you.
I also think it would be important to put all things on the table for discussion before finalizing how you'll proceed. Look at what came before and learn from it: what worked and what didn't work. What are the core elements that help define
Star Trek's appeal?
Allow me to speculate aloud. I recall when I saw the first pics of the tumbler, the new "batmobile" for Batman Begins. My first reaction was
WTF! Then I saw the film and saw how the vehicle was utilized and I changed my mind and clearly saw, "Yeah, now I get it and it makes perfect sense." And in context it made more sense than any batmobile that proceeded it. Still, other than the tumbler I still love the '60s Adam West batmobile even though it doesn't really make much sense.
And so in terms of a new
Enterprise I'd either want a tweaking of the original design for a mild reboot or I'd do something drastically different for a more clean sheet reinterpretation. I don't think it has to look exactly like the original
Enterprise to feel special to the crew and the audience. It's more about execution.
The appeal of a clean sheet approach is that you overcome the problems of how does it fit into established continuity simply because it isn't meant to mess with established continuity. This is what Abrams should have done openly without trying to be cute with stupid time travel and alternate timelines and whatever.

Part of the reason detractors dislike the film is the insistence of others trying to rationalize its existence in regards to established continuity. My (and some others) perspective is that it has nothing whatsoever to do with the original continuity and just accept it (or not) as such and as is, it's own thing and a new start. A clean reboot avoids this issue.