Developing optical processing tech is a LOT harder than making smaller chips to make smaller chips.
It requires creativity, a few geniuses along the way, billions in research money. And even then, it's not certain.
So far, the optical technology is FAR from being ready. And this will continue for some time to come.
I already metioned similar past examples.
Here's one - in the '60, everyone beleived that by now we'll have a very effective method of space propulsion. 50 years later, we still have chemical rockets. You see, developing such a revolutionary propulsion is a lot harder than the overly optimistic predictionns of times past would have one beleive.
As for the theoretical limit of development a civilization can reach - the laws of physics impose quite a few limitations.
For example - entropy, conservation of energy.
While I agree with you for the most part, you're ignoring any/all negative outside influences. Obviously, technological evolution can't happen in a vacuum, so it's impossible to say.
That millennium-plus long technological void you spoke of wasn't limited to just weaponry. It was across the board. The reason for this is because those in power had a vested interest in keeping the populous ignorant and in its place.
The only anomaly in the entire time period was the printing press and that was only "allowed" to succeed because it meant God's Word could then be spread to more people.
With rocket propulsion, it (despite eloquent Presidential speeches) was only promoted for its perceived military advantage. Once it was conduced that advantage wasn't practical or worth it, the only thing that remained was economics--in which there is nothing but negative results. For the people that matter, "research" for the sake of it is pretty low on the list. But be rest assured that if a resource that can be exploited on Mars is discovered the technology to get us there will increase ten-fold within a decade.
The internal-combustion engine is pretty obvious so I won't even bother. But for the record, research into making it more efficient as far back as the 80s were immediately squashed. Who's to say how far they would have come had they been allowed to evolve all this time?
Ironically, however, there is an opposite effect with computer-related technology. It's more "economically sound" to advance it forward as fast as possible--the initial cost is seen as insignificant. Faster, more advanced technology has allowed (forced?) the lemmings to do more work faster and more accurately. That is, of course, the work that isn't automated. Advanced AI can process limitless amounts of information in a very short time--and for next to no cost. So it's economically advantageous to obtain it as soon as possible. That's why it has advanced as fast as it has and will continue to do so.
Just look at its effect on society. The ability now exists to allow people to do work that at one-time had to be done on-site half-way across the globe.
But even with that that isn't. People are no longer incommunicado with their employers when they go home for the day, go on vacation, or what have you. The can be reached (and do their work) at any time and any place.
It's been suggested that people who get paid for a typical 40 hours or work a week, actually, on average, do closer to 60.
You think the people who matter aren't going to milk this cow for all its worth? If milking that cow means pushing the technology to the envelope as fast as possible, that's what will happen.