I'd argue we have no proof that Starfleet and the UFP didn't have advanced AI. They had pretty much sentient starship computers in TOS already - those just happened to know their place, and spoke in tinny, submissive voices. Starfleet in TOS already also knew how to build android bodies.
Sentient computers, yes... and quite spectacular failures at that; enough that it wasn't until the EMH that Starfleet even allowed mission critical duties to be carried out by one, even if it was just a CMO backup.
But sentience isn't the same thing as sapience: the ability to move and experience in a similar fashion as us, with the capacity for abstract thought and feeling (which, despite all his protestations, Data had in abundance). This doesn't seem like a great distinction, but just ask Sargon what it's like to be a brain-in-a-box and it's limitations...
Of course, the cybernetic aspects were likely easy. 400 years of research in limb replacement alone probably took care of that one, nevermind the various androids already found by the original
Enterprise. Of course, looking at androids like the Exo III and Mudd varieties, one can easily see their brains left much to be desired...
So perhaps Soong had to come up with something truly amazing, something completely finished, in the neglected field of positronics, before he could emerge from his hiding on Omicron Theta. Lore might have been it - but there was precious little time between the creation of Lore and the attack of the Crystalline Entity, probably not enough for staging a triumphant return.
How long a time, really, was it between Lore's creation and the Entity's attack? As far as I know, that's a very open question. Could have been weeks, years, or even a couple of decades. Soong's biography is suprisingly scant on these matters. In any case, I'm not sure if there was any evidence of there being 'little time' between the two events, though I could be wrong about that.
In any case, it would help if we knew what was so different about positronics that distinguishes it from duotronics, isolinear chips, or bioneural gel paks. In any case, even having had decades to study Data, Starfleet doesn't seem either interested or capable of reproducing the feat, and it still seems to be seen with a certain amount of awe... enough that I have to imagine it's somewhat superior to whatever computer tech they've already got going.
I'd argue because positronics was seen as a dead-end path - it was referred to as Soong's "dream", after all. Other sort of AI research was well respected, as we found in connection with Ira Graves. Perhaps UFP science could already do everything Soong's positronics promised to do... But Soong believed he could do it in more compact form with positronics, and a good demonstration of that would be to cram an entire AI into the skull of an android, when existing AIs were of mainframe size.
This is possible, though typically, once a feat is shown to be possible by scientists, the mere fact of it's proven possibility is enough to spur others to re-create it, usually faster, better and cheaper, especially with an example or prototype to look at. This doesn't seem to have happened with Data. Either the tech is
extremely complex, or it's advanced enough to defy reverse engineering. Either possibility could work, but unfortunately only raises even
more questions.
About 90% of colonies in Trek seem to be of isolationist mindset. They'd not trust outside authorities, least of all the soulless minions of orthodoxy from Starfleet. Hell, even the seemingly conformist major colony and former shipping hub Deneva didn't stay in any sort of steady contact with Earth or Starfleet - a silence of a full year was met with a shrug.
I'm sure Soong chose Omicron Theta specifically because he knew nobody there would squeal on him.
This I don't really have a reply for... other than this really should be investigated, as it doesn't seem like very healthy behavior to have isolationist colonies continually fleeing your sphere of influence. And while I understand that it's a Federation where worlds aren't forced to participate, it just seems ludicrous that all these colonies would choose to remain out of contact with each other... I know it's just a plot device in order to generate throwaway planets for the lates space terror of the week, but seriously, this is very weird behavior.
I see no evidence of an inability to reproduce or produce the hardware component of the tech. Data did it in a heartbeat in "The Offspring". There simply were fine details with the tech that people like Maddox didn't quite understand - there was no obstacle in manufacturing that we'd know of.
Whether hardware or software,
something is non-reproductable. Even Data couldn't pull it off, as Lal died of cascade failure days after activation, a problem he never seemed to find a solution for, and, in fact, Soong appears to be the only one who knows how to compensate for it, since according to Data himself, he shouldn't work (see his discussion of faith in 'Birthright'). Admittedly, I'm not sure what 'cascade failure' is, exactly, but it would appear to be the main hurdle for creating stable, positronic matrices that no one other than Soong has managed to work around.
But it's explicit Starfleet didn't even find the underground colony itself. Apparently the Tripoli folks just went there, thought "Hey, yet another colony of idiots completely wiped out by some random space threat, only this one left behind an android. Let's go, we have a dozen more cases like this waiting to be checked out this month!", and moved on.
If true (which I have no reason to believe it's not), that's just ridiculous. No wonder colonies drop by the truckload, if Starfleet forensics isn't even up to the task of finding out WHY colonies and people drop dead beyond saying 'Huh, would you look at that.' Not criticizing your point, which is valid, just that it's a dead-stupid way of handling things that I wouldn't expect from Starfleet. I can imagine how the plot of The Doomsday Machine would have gone if the
Enterprise had adopted that attitude...
That sounds relatively natural. Omicron Theta was a non-entity as far as Starfleet and the UFP were concerned; they had few obligations regarding it, and few quarrels. At that point, they apparently didn't have evidence of a pattern of destruction relating to what they saw at OT, either, so the "random space threat" would warrant no further study, at least not with any urgency.
My only problem with this logic is that we saw exactly this kind of investigation going on ALL THE TIME in the show. Or is the Enterprise the only ship with the personnel capable of investigating such things (must have been, since they were the one's that figured out what happened with Omicron Theta... thirty some odd years later).
I'd think the use of a human template was only a feature he added later on, for the android copy of Tainer he built after the original's death. And presence or lack of emotions would be relatively simple fiddling: holocharacter designers probably do that all the time, and they have been around for a long time (Janeway had holonovels in her childhood already).
We don't know anything about the ease or difficulty of holoprogramming, whether the 'personalities' are coded by hand or the result of some kind of synaptic scanning or neural emulation or some such. As well, we don't know how personalities are 'customized', though I suspect it's not easy... otherwise, Voyager could have saved itself some headaches by just saying 'Decrease EMH snarkiness by 90%'.
As for human emulation being a later addition, I don't see why. It's obvious that Tainer's 'rebirth' was an application of Ira Graves synaptic scanning technique that the old man himself used to hijack Data's body. Soong was obviously familiar with it (it seems too much of a coincidence that he would have developed it independently, especially seeing as the two worked together at some point in the past), and, I suspect, used a more primitive understanding/application of it with both Lore and Data, not tranferring his memories or personality, neccesarily, but possibly his basic brain patterns or some such as a 'template' on which to build, which makes more sense than re-crafting every nuance of human experience from scratch, essentially re-inventing the wheel. Personally, I'd always wondered why, if Soong could eventually pull off an entire tranfer with Juliana, why he never did it to himself. The man certainly had the mindset to want to live forever... you'd think he'd have done it.
This begs the question of whether the young Soong really was a visually identifiable celebrity or not. Considering his shady and elusive nature, he might even have changed his appearance after disappearing from public but before building his androids. However, ENT sort of negates this possibility by showing us a Soong ancestor with the same looks and the same megalomania.
But considering his megalomania and narcissism, I highly doubt he'd change his appearance... that just doesn't seem in character. And why, really? He wasn't a criminal, just a disgrace. Obviously, he wouldn't be a celebrity to the 'man on the street'. but I imagine to his peers and contemporaries at Daystrom, he'd be recognizable, like Henry Ford knowing the face of Nikola Tesla.
Good points all, and an interesting discussion.