The video began with a clip of a foreign-language movie dubbed to English, but never showed any deepfaked examples in the same direction (they were all films in English dubbed to other languages)....
The video began with a clip of a foreign-language movie dubbed to English, but never showed any deepfaked examples in the same direction (they were all films in English dubbed to other languages). I’m not as familiar with reading lips in German or Japanese so it’s harder for me to vouch for their accuracy. I would’ve liked to see some demonstrations of the tech producing lipsyncs for my own language.
There’s a good comment on Gizmodo pointing out that often the dubbed voice performance doesn’t even sync up well with the video in the first place. Maybe the translated line just takes longer (or is quicker) to deliver than the original, screwing up the timing. Or the voice actor used a tone that doesn’t align with the original actor’s body language. I feel like these things are easily dismissed when the movie is obviously dubbed, but when deepfaked may end up in a sort of uncanny valley that actually takes you more out of the scene.
I’m also curious what legal issues might arise from actors challenging the sale of these manipulations of their prior performances.
I honestly hope this becomes a roadblock to this, because I really don't want this to catch on. I get that having the lips match what's on screen will make dubs much less painful to watch, but at...
I’m also curious what legal issues might arise from actors challenging the sale of these manipulations of their prior performances.
I honestly hope this becomes a roadblock to this, because I really don't want this to catch on. I get that having the lips match what's on screen will make dubs much less painful to watch, but at the same time I still have problems with dubs on a conceptual levels. No matter how you cut it, you're creating an adulteration of the creators' original intent. So if you watch Parasite that has been seamlessly transformed to feature English speech, you are still watching something very much like it, but you still haven't seen Parasite.
We are not talking about accessibility; that's beyond the scope of what I'm talking about. Frankly, I find your pedantry to be somewhat insulting. The people making the movie know that not...
We are not talking about accessibility; that's beyond the scope of what I'm talking about.
Frankly, I find your pedantry to be somewhat insulting. The people making the movie know that not everyone's going to have the same viewing experience. The reason why audio mixing has changed from discrete location-based channels to virtual sound sources that get remixed locally on-the-fly is because they understand that not every cinema is going to have the same kind of experience. But in any case, the viewer is still able to see fairly clearly what the creative intent was. This is much deeper than the "wow effect" that can be afforded by higher-end setups.
To get to the root of our disagreement, there are many different places were one can draw the line as to what the best way to view a movie is. All I am doing is explaining why I draw mine where I do.
As a side note, I find myself somewhat amused at your comparison to Christianity because if we could bring Paul back and explain what Christianity is today, I'd imagine he'd weep. Not only is the church he founded completely different than how he left it, the world is full of impostors who further sully the image he has of his god.
Fine, my reaction to this technology was extreme. I'll take that back. Honestly, I wish that you had explained why that was important to you instead of trying to say that my opinions were wrong,...
Fine, my reaction to this technology was extreme. I'll take that back.
Honestly, I wish that you had explained why that was important to you instead of trying to say that my opinions were wrong, because it was only after doing some research that I began to understand your position. I found out that the 2.5% quantile of readers are capable of reading 161 words per minute - more than fast enough to read subtitles - but then I found the rate of people in the US who are illiterate is 21%, and that is a lot of people. I didn't think that there was actually a large percentage of people who were left out by subtitles.
In the meanwhile I honestly don't think that you're all that interested in understanding my opinion so I'll just leave things here.
It isn't only that people are too slow for subtitles. They're also distracting. And if you don't understand the language at all, more often than not, they're insufficient to give a good experience...
It isn't only that people are too slow for subtitles. They're also distracting. And if you don't understand the language at all, more often than not, they're insufficient to give a good experience to most people. I'm not going to listen to japanese audio just because it matches the director's intent and the lip movements. I listen because I want to hear information about the plot.
Without dubs, most people wouldn't watch US movies at all here in Germany.
Now granted, to a purist or a fan of the director, or in some cases where language does a lot of work (Hans Landa in inglorious bastards speaking different languages e.g.) I certainly agree that you want the original, and I accept that. Dubs to German are good but I often find myself discovering minutia that were lost in translation when I watch the original afterwards. In that way I can see your stance.
With that logic, can you really see Parasite if you don't know Korean? There are always going to be cultural references, figures of speech, tones and implications that don't translate. I feel like...
With that logic, can you really see Parasite if you don't know Korean? There are always going to be cultural references, figures of speech, tones and implications that don't translate. I feel like "Korean" -> "English subtitles" is a bigger jump away from the creators original intent than "English subs" -> "English dubs" is.
Is there a particular reason why you want this to fail, rather than just abstaining? Because I don't think subtitles are going away due to this. With video on demand these days, you can just watch...
Is there a particular reason why you want this to fail, rather than just abstaining? Because I don't think subtitles are going away due to this. With video on demand these days, you can just watch the version you want with the subtitles you want. If I'm on netflix, I don't have to watch the German dubs, I can just watch in English and add subtitles if the actors are hard to understand. I would imagine this wouldn't be any different if this caught on. I could just watch the original, if that's what I wanted and add some subtitles.
I prefer subtitles, but completely disagree with your way of thinking. Dubs will exist either way, just because you prefer people don’t use them doesn’t mean they shouldn’t be allowed to improve...
I prefer subtitles, but completely disagree with your way of thinking. Dubs will exist either way, just because you prefer people don’t use them doesn’t mean they shouldn’t be allowed to improve the viewing experience for those that choose to watch it that way.
I can go really deep into this subject, but I'm going to try to keep it as simple as I can for the time being. I have two different ways I enjoy a film. One is for the story; you watch the film...
I can go really deep into this subject, but I'm going to try to keep it as simple as I can for the time being.
I have two different ways I enjoy a film. One is for the story; you watch the film because it tells you something important; it has a message that it wants to impart to you. Dubs, as they are now, are capable of doing this to varying degrees of success, and honestly speaking I think that this kind of tool will push that degree towards better levels of success.
The second way I enjoy film is for the artistry, and that's where dubbing falls apart. Right now dubbing obliterates your understanding of the actors' performances, and that's the main aspect that affects the film's storytelling success. The performance of the dubbers are also not subject to the vision of the original director, which is part of why dubs often have 'errors' of inflection and pacing. Dubbing also has the effect of echoing into the other aspects of the film. The audio engineers who mixed the original film will have their creative choices eliminated. Thankfully the cinematography stays the same, but because dubs require people speaking at different cadences and lengths, it can also affect your perception of how the film was edited.
I agree with this. Language (and acting) is more than speech and lips, and syncing up lips is only part of it. The thing that really got me thinking though, is that now that the tech is at a...
I agree with this. Language (and acting) is more than speech and lips, and syncing up lips is only part of it.
The thing that really got me thinking though, is that now that the tech is at a fairly accessible point, how does that alter film making? If the actors/director/etc know and plan for the ability to do this, how can the performances change to make the dubbed result less uncanny?
Am I just seeing things or is there some visible warping on all those examples? Especially that one clip of de Niro, it looked like there was quite a bit of warp artifacting around his jowls.
Am I just seeing things or is there some visible warping on all those examples? Especially that one clip of de Niro, it looked like there was quite a bit of warp artifacting around his jowls.
I'm seeing it to. Mouths are one of the hardest things to "fake" or alter. Tons of studies have shown that our brains are so attuned to reading facial features that we can easily see if something...
I'm seeing it to. Mouths are one of the hardest things to "fake" or alter. Tons of studies have shown that our brains are so attuned to reading facial features that we can easily see if something is off. It sends off signals in that lizard part of our brain that says "something isn't right here".
Two Minute Papers put up a video today that's related to this topic: AI makes near-perfect deep fakes in 40 seconds. It focuses a lot on faking lip movements and altering the audio to match, and...
Two Minute Papers put up a video today that's related to this topic: AI makes near-perfect deep fakes in 40 seconds. It focuses a lot on faking lip movements and altering the audio to match, and it's a hell of a lot better than these examples. It can even adjust the force/verve behind the fake and make a convincing fake with a mere 30 seconds of subject audio/video data.
For All Mankind used deep fakes for Reagan and a few other dead celebrities. It looks really convincing, but it's shown through the lens of an older CRT TV.
The video began with a clip of a foreign-language movie dubbed to English, but never showed any deepfaked examples in the same direction (they were all films in English dubbed to other languages). I’m not as familiar with reading lips in German or Japanese so it’s harder for me to vouch for their accuracy. I would’ve liked to see some demonstrations of the tech producing lipsyncs for my own language.
There’s a good comment on Gizmodo pointing out that often the dubbed voice performance doesn’t even sync up well with the video in the first place. Maybe the translated line just takes longer (or is quicker) to deliver than the original, screwing up the timing. Or the voice actor used a tone that doesn’t align with the original actor’s body language. I feel like these things are easily dismissed when the movie is obviously dubbed, but when deepfaked may end up in a sort of uncanny valley that actually takes you more out of the scene.
I’m also curious what legal issues might arise from actors challenging the sale of these manipulations of their prior performances.
I honestly hope this becomes a roadblock to this, because I really don't want this to catch on. I get that having the lips match what's on screen will make dubs much less painful to watch, but at the same time I still have problems with dubs on a conceptual levels. No matter how you cut it, you're creating an adulteration of the creators' original intent. So if you watch Parasite that has been seamlessly transformed to feature English speech, you are still watching something very much like it, but you still haven't seen Parasite.
We are not talking about accessibility; that's beyond the scope of what I'm talking about.
Frankly, I find your pedantry to be somewhat insulting. The people making the movie know that not everyone's going to have the same viewing experience. The reason why audio mixing has changed from discrete location-based channels to virtual sound sources that get remixed locally on-the-fly is because they understand that not every cinema is going to have the same kind of experience. But in any case, the viewer is still able to see fairly clearly what the creative intent was. This is much deeper than the "wow effect" that can be afforded by higher-end setups.
To get to the root of our disagreement, there are many different places were one can draw the line as to what the best way to view a movie is. All I am doing is explaining why I draw mine where I do.
As a side note, I find myself somewhat amused at your comparison to Christianity because if we could bring Paul back and explain what Christianity is today, I'd imagine he'd weep. Not only is the church he founded completely different than how he left it, the world is full of impostors who further sully the image he has of his god.
Fine, my reaction to this technology was extreme. I'll take that back.
Honestly, I wish that you had explained why that was important to you instead of trying to say that my opinions were wrong, because it was only after doing some research that I began to understand your position. I found out that the 2.5% quantile of readers are capable of reading 161 words per minute - more than fast enough to read subtitles - but then I found the rate of people in the US who are illiterate is 21%, and that is a lot of people. I didn't think that there was actually a large percentage of people who were left out by subtitles.
In the meanwhile I honestly don't think that you're all that interested in understanding my opinion so I'll just leave things here.
It isn't only that people are too slow for subtitles. They're also distracting. And if you don't understand the language at all, more often than not, they're insufficient to give a good experience to most people. I'm not going to listen to japanese audio just because it matches the director's intent and the lip movements. I listen because I want to hear information about the plot.
Without dubs, most people wouldn't watch US movies at all here in Germany.
Now granted, to a purist or a fan of the director, or in some cases where language does a lot of work (Hans Landa in inglorious bastards speaking different languages e.g.) I certainly agree that you want the original, and I accept that. Dubs to German are good but I often find myself discovering minutia that were lost in translation when I watch the original afterwards. In that way I can see your stance.
With that logic, can you really see Parasite if you don't know Korean? There are always going to be cultural references, figures of speech, tones and implications that don't translate. I feel like "Korean" -> "English subtitles" is a bigger jump away from the creators original intent than "English subs" -> "English dubs" is.
Is there a particular reason why you want this to fail, rather than just abstaining? Because I don't think subtitles are going away due to this. With video on demand these days, you can just watch the version you want with the subtitles you want. If I'm on netflix, I don't have to watch the German dubs, I can just watch in English and add subtitles if the actors are hard to understand. I would imagine this wouldn't be any different if this caught on. I could just watch the original, if that's what I wanted and add some subtitles.
I prefer subtitles, but completely disagree with your way of thinking. Dubs will exist either way, just because you prefer people don’t use them doesn’t mean they shouldn’t be allowed to improve the viewing experience for those that choose to watch it that way.
I can go really deep into this subject, but I'm going to try to keep it as simple as I can for the time being.
I have two different ways I enjoy a film. One is for the story; you watch the film because it tells you something important; it has a message that it wants to impart to you. Dubs, as they are now, are capable of doing this to varying degrees of success, and honestly speaking I think that this kind of tool will push that degree towards better levels of success.
The second way I enjoy film is for the artistry, and that's where dubbing falls apart. Right now dubbing obliterates your understanding of the actors' performances, and that's the main aspect that affects the film's storytelling success. The performance of the dubbers are also not subject to the vision of the original director, which is part of why dubs often have 'errors' of inflection and pacing. Dubbing also has the effect of echoing into the other aspects of the film. The audio engineers who mixed the original film will have their creative choices eliminated. Thankfully the cinematography stays the same, but because dubs require people speaking at different cadences and lengths, it can also affect your perception of how the film was edited.
I agree with this. Language (and acting) is more than speech and lips, and syncing up lips is only part of it.
The thing that really got me thinking though, is that now that the tech is at a fairly accessible point, how does that alter film making? If the actors/director/etc know and plan for the ability to do this, how can the performances change to make the dubbed result less uncanny?
Am I just seeing things or is there some visible warping on all those examples? Especially that one clip of de Niro, it looked like there was quite a bit of warp artifacting around his jowls.
I'm seeing it to. Mouths are one of the hardest things to "fake" or alter. Tons of studies have shown that our brains are so attuned to reading facial features that we can easily see if something is off. It sends off signals in that lizard part of our brain that says "something isn't right here".
Two Minute Papers put up a video today that's related to this topic: AI makes near-perfect deep fakes in 40 seconds. It focuses a lot on faking lip movements and altering the audio to match, and it's a hell of a lot better than these examples. It can even adjust the force/verve behind the fake and make a convincing fake with a mere 30 seconds of subject audio/video data.
For All Mankind used deep fakes for Reagan and a few other dead celebrities. It looks really convincing, but it's shown through the lens of an older CRT TV.