Using Deepfake in Trek... Yes or no?

[Pauln6]

Arrrgh. As usual, I have hit a stumbling block with Wav2Lip. The Google Collab links don't install but don't give me an error message on which I can work (the python installation just hangs - all the more annoying since I know I have the right version of Python already installed but possibly not in the right folder and it won't complete the next step without completing this one.

So I figured I would try a manual install on my laptop since my GPU is powerful enough to cope. Github can be a bit impenetrable if you have no training in coding. They rarely spell out the process in laymans terms (I think because they want to discourage the average member of the public from abusing the technology. The manual install just says 'clone the repository' but the commands listed are not the commands you need to do that. It then says save the pre-trained models and gives you a file name but doesn't tell you WHERE to save them. You watch the help videos and they say, 'this took 10 minutes' but after 4 hours I am barely at stage 2. I give up.

EDIT: Obviously, I didn't give up. I managed to work out the collab issue was a Google docs path error because they'd taken the space out of My Drive. Sigh. It still isn't working in that it's just hanging without producing anything but I at least I don't feel defeated by stupid syntax.

I still have no idea how to manually install it...

EDIT 2: Ok - eventually got it to run but only with really low resolution. One layer of Deepfake later and the results are in. I tried to mask everything but her face using layers. The lipsync is improved. i will have to see if the lower resolution is that noticeable on a bigger screen.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

I did run it for her other scene but looking at the resolution on this one, I can't see that it's going to look as good on more of a closeup. I tried to run it Kirk's space suit as well so the lipsync for "I have him in sight" syncs better with, "Just like old times." but the image was too large. I ran this image by cropping out Kirk, so it might work if I crop everything but his face. Worth a try next time I have a bit of time.

Edited: Swapped for a video that shows how I managed the process.

 

[Scionz]

Generally no, but it should be used to help enhance any kind of natural flashback where characters appear younger. For example, Data in ST:P is infinitely helped by Deepfakes. It doesn't replace the make up and CG. It works along side it.

 

[Pauln6]

Moving on from Rand, I did a quick experiment with Wav2Lip to see if it might be possible to incorporate one of the jokes from the In Thy image script into my fan edit. I can probably improve on this substantially but it's not terrible for as little as 2 hours work!

 

[jackoverfull]

Nice. The voice sounds quite robotic, though, especially in the first part of the sentence.

 

[Pauln6]

Nice. The voice sounds quite robotic, though, especially in the first part of the sentence.

Yeah the vocoded dialogue varies in quality. I can keep rerunning it but it is a struggle to get the intonation where you want it. Ironically, in the one that sounded the least robotic, she says 'everything' in such a way that she sounds like she genuinely does want to watch them have sex.

 

[publiusr]

Now one plus of advancing tech is to put motion in old photos, as in a recent phys.org blurb. Early attempts to “morph” the different frames of the Mt. St. Helens eruption were crude…but it will get better

 

[Rhodo]

I prefer CGI to deepfake, as it's more interesting to me and involves artwork/sculpture - but I'm aware not many people agree, or find resurrecting the image of deceased performers to be distasteful.

Some are saying to recast rather than deepfake or similar. I say don't even do that - strike out in a direction and make something completely different. That way, we can pay actors IRL to originate iconic roles, the way Trek has done consistently well for years.

 

[fireproof78]

I say don't even do that - strike out in a direction and make something completely different. That way, we can pay actors IRL to originate iconic roles, the way Trek has done consistently well for years.

Agreed. Stop living in the past.

 

[jackoverfull]

Amen to this. But the word in the Hollywood of today is nostalgia and so here we are, with cameos, recastings and deepfakes in sequels and prequels. Perhaps one day this will pass, hopefully.

 

[SolarisOne]

Nah, just recast. Cheaper, and ultimately less distracting. Never say never though.

Or create new characters.

 

[WarpTenLizard]

Not unless its' a hologram, android, or grainy recording the other characters are watching. In other words, something that's artificial in-universe.

I didn't have a problem with Data in "Star Trek: Picard," because he's not supposed to look human. And frankly, looking less human is more believable for his character. So likewise, I'd be happy to see Robert Picardo de-aged as the EMH, especially if filtered through the new-Trek style of holograms, to hide any flaws in the special effect.

But trying to use that for a regular human character? I know the MCU somehow did it, perfectly, with several actors. But I don't want anything like Tarkin in "Rogue One," with his strange goldfish mouth.

 

[Sisko_is_my_captain]

Actually, I think it would be interesting to deep fake anybody that a changeling was impersonating. Rather than just use the unaltered actor for the character being assumed by the changeling, deep fake the character (perhaps even over the actual actor's face) just to achieve a slight uncanny valley effect. It is supposed to be a puddle of goo recreating a human face. It should look slightly wrong, but you shouldn't be able to put your finger on what is off.

Even better, deep fake every scene with Odo. Just make him that much more unnatural.

 

[Pauln6]

I bought the new Star Trek boxed set and had another go at deepfaking Grace Lee Whitney for my homage scene, using the cleaned up, brighter footage from the Bluray. It still needs a bit of work (I need to separate the dialogue from the soundtrack so I can slow down the music across the whole scene, I need to try and extend her reaction shot fractionally as I think it cuts away too quickly, and I need to marry up her epaulettes and right arm with the rest of the image a bit better) but is much better than my first attempt.

This has taken a huge amount of man hours for 4 seconds of screen time, and it still needs a lot of work. I'm not sure why anybody would want to do this as opposed to re-casting for anything other than a brief nostalgic cameo.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[Yistaan]

I bought the new Star Trek boxed set and had another go at deepfaking Grace Lee Whitney for my homage scene, using the cleaned up, brighter footage from the Bluray. It still needs a bit of work (I need to separate the dialogue from the soundtrack so I can slow down the music across the whole scene, I need to try and extend her reaction shot fractionally as I think it cuts away too quickly, and I need to marry up her epaulettes and right arm with the rest of the image a bit better) but is much better than my first attempt.

This has taken a huge amount of man hours for 4 seconds of screen time, and it still needs a lot of work. I'm not sure why anybody would want to do this as opposed to re-casting for anything other than a brief nostalgic cameo.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

Wait, am I missing something? Why does Grace need to be deepfaked into a movie she's already in? I thought deepfaking for Trek movies would be reserved for, say, Patrick on Shinzon in Nemesis (been done badly on youtube already), Kirstie Alley on Robin Curtis or the other way around for Saavik, or pasting Glenn Corbett into First Contact.

 

[Pauln6]

Wait, am I missing something? Why does Grace need to be deepfaked into a movie she's already in? I thought deepfaking for Trek movies would be reserved for, say, Patrick on Shinzon in Nemesis (been done badly on youtube already), Kirstie Alley on Robin Curtis or the other way around for Saavik, or pasting Glenn Corbett into First Contact.

Long story but I always wanted to give Grace a cameo in the second half of the movie, ideally a quick shot of her staring dumbfounded into the cloud during the fly through. Unfortunately, that proved beyond my abilities at that time so I got sucked down a rabbit hole where I re-introduced the Memory Wall scene that required me to insert a CGI avatar of Grace inside Spock's helmet. It's minimal onscreen dialogue but I needed to put a layer of deepfake over the CGI.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[Tim Thomason]

Spoiler: Prodigy Ep 6

With the latest episode of Prodigy, we get a curious "deepfake" of reusing voice clips (not well-edited, it seems) for Leonard Nimoy, Nichelle Nichols, René Auberjonois, Gates McFadden (also some real voiceover), and James Doohan. Three out of five of whom are deceased. I see lots of praise for the scene, as paying homage to the decedents, rather than being disrespectful. Maybe because it's not a "real" deepfake (stylized animation and old voice clips), and its set on the Holodeck with fake characters.

 

[jackoverfull]

Honestly, I was thinking of this thread watching it. I like what they tried to do but the results were far from being smooth.

 

[seigezunt]

I think think it's likely, but the technology isn't quite there yet.
It would be a great way to bring back some legacy characters.
Seeing the recent excursion reminded me of the old Bring Back Kirk video, which was a lesson in how compelling the option is, and terrifying at the same time.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[jackoverfull]

Loved BBK.

Regarding the technology not being there yet…I think mostly it is, but not on the budget of a the show. Give it a few years, though…

 

[Yistaan]

Loved BBK.

Regarding the technology not being there yet…I think mostly it is, but not on the budget of a the show. Give it a few years, though…

Whatever happens on the fan level with deepfakes, CBS will stop it dead in its tracks, and that's not even getting into the whole likeness used without permission complaints that will be filed by actors and their estates (apparently Scott Bakula was rumored to be extremely protective of his likeness rights even when he would have been paid legitimately, thus supposedly why Archer and Enterprise were hardly ever featured in comic books).