TMP Deepfake

Discussion in 'Star Trek Movies I-X' started by Pauln6, Dec 12, 2018.

  1. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    I thought I'd bite the bullet and download a face swap program to experiment with images of TOS Rand. Three days in and it's still learning with a discrepancy of 0.036 (apparently I have to wait for it to get below 0.020). I had to lower the settings as my laptop doesn't have enough working memory so I'm expecting the results to be a bit below par but if it's passable then maybe I'll have another crack after I've upgraded.

    As an aside, Grace felt she was singled out by Wise because she played a joke on him with Roddenberry and she wasn't allowed to wear any make-up in her close up scenes, not even eye liner. When you face swap, you really notice what a difference eyeliner makes to her appearance.
     
  2. Maurice

    Maurice Snagglepussed Admiral

    Joined:
    Oct 17, 2005
    Location:
    Real Gone
    The main problem with that sequence is it's utterly tone-deaf the the original film's style. Lucas shot with a "documentary camera" aesthetic which eschewed the camera movements and framings throughout this. Performance and athletics aside, it's as out of place in Star Wars as Bonnie and Clyde's finale would be in a 30's Warner Bros. gangster pic.
     
    Kor, BeatleJWOL, J.T.B. and 1 other person like this.
  3. jaime

    jaime Vice Admiral Admiral

    Joined:
    Sep 2, 2013
    Location:
    London
    I dunno. Original film...yeah maybe...but as a halfway house to knit in with the Kenobi of the prequels, it’s certainly working. Needs to be just a little more ‘kenobi could win this, but is choosing not to’ and maybe dial back the music to be more Williams. Fifty fifty in the sound mix flashbacks. But...I wouldn’t have minded a scene like this back when they added the blinking evokes and what not. It makes it tonally closer to the prequels and helps smooth the joins. I did like the fire on Vader’s arm.
    Maybe there should just be a hint that the inner Anakin was holding back a tiny bit, a hint of obi wan trying to see if anything was left of his friend. I think it works.

    The rand in TMP thing...not so much tbh.
     
    Pauln6 likes this.
  4. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    I've never tried experiments like this before so Rand was a good starting point for proof of concept: plenty of full frontal, limited TOS dialogue to wade through, only a couple of shots where she could logically fit in, plus I have a soft spot for her.

    If I had the skills and equipment of whoever made the fight scene above, who knows what monstrosities I could create!

    That said, I have now looked through some of the other dialogue and it's a real challenge to find any that fits well.

    Part of me would like to add some gallows humour to the V'ger cloud scene but I doubt I can find anything suitable.
     
  5. jaime

    jaime Vice Admiral Admiral

    Joined:
    Sep 2, 2013
    Location:
    London
    I think rand (soft spot aside) got basically as much as most of the non-triumvirate characters. I don’t remember seeing Sulu doing anything but smile.
     
  6. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    Actually Sulu gets a fair amount of subtle stuff that is easily overlooked but he is at least a presence on the bridge and in command on two occasions, whereas Rand doesn't man a station on the bridge or in engineering. I'd love it if I could give every character one extra line though. I fiddled with some dialogue but it sounds terrible stitched together. Uhura has some great lines in TOS and in STIII but it's hard to find something good to go alongside them.
     
  7. Maurice

    Maurice Snagglepussed Admiral

    Joined:
    Oct 17, 2005
    Location:
    Real Gone
    My point is simply that If you stuck that scene into the film proper it would not work for the reasons stated. As a stand-alone it works okayish.
     
    BeatleJWOL and Serveaux like this.
  8. jaime

    jaime Vice Admiral Admiral

    Joined:
    Sep 2, 2013
    Location:
    London
    I think it would work. Especially if another bit here or there was jazzed up a little (as they have been) if the film is then in context as ‘episode iv’ basically. But only in that context.
     
  9. Serveaux

    Serveaux Fleet Admiral Premium Member

    Joined:
    Dec 30, 2013
    Location:
    Among the sellers.
    Nope. Maurice is right.
     
    Firebird and BeatleJWOL like this.
  10. BeatleJWOL

    BeatleJWOL Commodore Commodore

    Joined:
    Feb 21, 2012
    Location:
    Winston-Salem, NC
    Yup. There are pieces of it that would spice up the scene as presented but the whole thing is about as fanwanky as it gets.
     
  11. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    I think that different people like different things in their movies. I mean, somebody actually thought it was important for Greedo to shoot first.

    I wouldn't hate this scene but I don't think it's necessary. Similarly, I prefer STII without the lengthy homage to Laurel and Hardy titled droids on conveyor belt.
     
  12. jaime

    jaime Vice Admiral Admiral

    Joined:
    Sep 2, 2013
    Location:
    London
    Nope.
    But you agree with him.
    Opinions.

    I dont think it would work in ANH by itself, but when put in context next to the prequels, i think it works. Because this then becomes the equivalent of Lukes salt planet fight in TLJ.
     
  13. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    Deepfake sucks so bad on my not very bright laptop. Worse, I had to reboot on Wednesday and when I restarted the learning process, none of the data re-loaded so I had to start from scratch. I don't think I can commit to running the program for as long as if takes to work if I can't even switch off the laptop. I'm already bored of waiting for it to hit it's 0.02 variance. I'm thinking I would rather wait until I have a PC with a graphics card when the process only takes about 72 hours.

    As a final experiment I've loaded on Terrence O'connor's likeness. She's the only obvious choice to be replaced by Rand in the second half of the movie so I might as well see how it looks.
     
    Last edited: Jun 23, 2019
  14. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom


    Sixth attempt at lip syncing an extra line into Rand's scene with McCoy. Tiny, but it triples her dialogue in the scene. Baby steps. This wasn't deepfake mind you, this was done manually by pasting her lip movements from the earlier scene - challenging given how little on screen dialogue she has in the movie.
     
    Last edited: Jul 24, 2019
  15. Maurice

    Maurice Snagglepussed Admiral

    Joined:
    Oct 17, 2005
    Location:
    Real Gone
    I am right because a film has to be stylistically cohesive, and that sequence is completely at odds with the look of the entirety of the balance of the film's live-action. Like a generic "flesh" colored band-aid on a dark-skinned person, it would stick out like a sore thumb.

    BTW, in June I saw a gorgeous 1985 70mm print of Star Wars with no Special Edition nonsense in it and the whole audience cheered when Han blows Greedo away. :D
     
  16. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    There is a line to be struck and you can't please all of the people all of the time. I find that I like 70% of the CGI in the remastered TOS less than the original.
     
    jaime likes this.
  17. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom


    My first upgrade to the Rand scene in Kirk's quarters importing a close up of Grace using layers. It will be challenging to manually splice the dialogue because of the shadows on the neck. A few more passes to add more transparency might work. Erasing the transporter room will be even more of a challenge because the background is so dark compared to the quarters. I suspect I will end up with inconsistencies around her hair, which are only obvious when the film is running.

    Still it's an interesting (for me) challenge. I think I'll revisit and edit some frames later in the year.
     
    Haggis and tatties likes this.
  18. Maurice

    Maurice Snagglepussed Admiral

    Joined:
    Oct 17, 2005
    Location:
    Real Gone
    Her eyeline is all wrong for the screen. It looks like she's looking at Bones.

    I can't even understand what she's saying after the galley bit.
     
  19. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom
    Yeah there are serious limitations when you only have one head shot to work with, although McCoy does respond so eyeline is not a deal breaker at this early stage. The longer shot had the right eyeline but doesn't look as clear and crisp. I could move her further to the right but I'm not sure that will help much.

    I can't remove the music track fully which is why there is some background noise at this stage. The line from Miri was whispered so it is hard for us old 'uns to hear clearly. I have to make sure I have no background music on when I'm checking for lip sync.
     
    Last edited: Aug 6, 2019
  20. Pauln6

    Pauln6 Rear Admiral Rear Admiral

    Joined:
    Jul 24, 2009
    Location:
    Bristol, United Kingdom

    Still playing with this. My laptop has sort of plateaued on the Deepfake. I think it's too stupid to run through a big enough sample size to improve the image so this is hybridised with the original footage. It's shame because the Deepfaked eybrows move more naturally with the speech but they are too blurry at the moment. I'll keep running it occasionally in the background and see how far I can take it.

    The Deepfake just doesn't seem to cope with the close up image in the quarters (lip sync looks jerky and she never seems to close her mouth). I think maybe I should go back to the smaller image where her eyeline is correct as Maurice pointed out. I could paste some lip sync from TOS and then run Deepfake to make it look more natural. Smaller image but a bigger job though.

    Terrence O'Connor is safe though. She spends too much time with her face to the side and the program is more likely to paste over Scotty. It isn't a good look.
     
    Last edited: Oct 13, 2019