Scott Mann’s problem was that he used too many f-bombs.
During the making of “Fall,” the writer-scary director’s thriller about rock climbers stuck on top of a remote TV tower, he told the two main characters to have fun with their lines. That improv got 35 “f-cks” into the movie, solidly putting it in R-rated territory.
But when Lionsgate agreed to sell “Fall,” the studio asked for a PG-13 version. To clean up the movie, all but one of the bad words would have to be taken out.
“How do you find a solution?” Mann remembered this in October, two months after the movie came out, from the glass-walled conference room of his office in Santa Monica. He had taken a prop vulture from the set and set it up in the lobby.
After all, reshoots are expensive and take a lot of time. Mann said that he had filmed “Fall” on a mountaintop, where he had to deal with COVID, hurricanes, and lightning storms. At one point, a colony of fire ants moved into the main set of the movie, a 100-foot-long metal tube. When the crew woke them up, the swarm covered the set “like a cloud.”
Mann said, “I think ‘Fall’ was the hardest movie I’ve ever made.” Could he avoid doing it again?
He realized that the answer might be a project he’d been working on at the same time as the movie: software with artificial intelligence that could edit footage of the actors’ faces long after the main filming was done, changing their expressions and mouth movements to match the newly recorded dialogue.
It seems like a simple way to use a technology that experts say is about to change almost every part of Hollywood, from the way people work and make money to how people think about what’s real and what’s fake.
Robert Wahl, an associate professor of computer science at Concordia University in Wisconsin who has written about the ethics of CGI, said in an email that AI will do for moving pictures what Photoshop did for still ones. “We can’t believe everything we see anymore.”
A piece of software to help with dubious dubs
Mann got into the world of software after he worked with Robert De Niro on a project that didn’t go well.
Mann’s 2015 crime thriller “Heist” had De Niro in it, and the two of them had put a lot of time and thought into the well-known actor’s role. But when it came time to make changes to the movie so it could be shown in other countries, Mann said he wasn’t happy.
When movies are shown in other countries, the dialogue is often redone in a different language. This is called “dubbing,” and it makes the movie available all over the world. However, it can be shocking to see an actor’s mouth move out of sync with the words they’re supposed to be saying. One common solution is to rewrite the dialogue so that it fits better with the existing visuals. However, these changes sacrifice the original vision of the creative team in order to make the story easier to understand.
Mann said of the dubs, “All the subtle things I’d worked out with Robert De Niro were now different.” “I felt pretty bad about it.”
“Final Score,” the next movie he worked on, only made things worse. Mann tried scanning the heads of his actors so he could better sync up their speech, but it was too expensive and made the movie look strange.
The fan of visual effects didn’t find a possible solution until he looked for new ideas. He found a 2018 academic paper that talked about neural networks, which are computer programs that try to mimic the structure of a brain. These programs tried to put one actor’s facial expression on another actor’s face.
Fascinated, Mann contacted the paper’s authors and started working with some of them on a simple “vubbing” tool, which means dubbing with images instead of sounds. After that, a friend of a friend named Nick Lynes, who had experience with online games, joined the team. This gave them a foothold in the tech sector as well.
Together, people from three very different fields—film, science, and the software business—built Flawless, an artificial intelligence (AI) filmmaking company with offices in both Santa Monica and London.
In a broad sense, the company’s technology can find patterns in an actor’s phonemes (the sounds they make) and visemes (how they look when they make those sounds) and then update the on-screen visemes to match when new phonemes are recorded. The company’s “fix for film dubbing” was named one of the best inventions of 2021 by Time magazine.
The rush to remove dozens of f-bombs from “Fall,” on the other hand, raised a question that could have much bigger repercussions: could Flawless change what the characters said, not just how they said it?
Mann, who lives in Los Angeles, said, “We took the actresses to a recording studio in Burbank and said, ‘Okay, here are the new lines.'” Then they plugged the new sound into the dubbing software, which changed the faces of the actors on screen to match.
“We added the shots, and then the MPAA looked at it again and gave it a PG-13 rating,” he said.
A few weeks after the movie came out, Mann sat in his conference room in Santa Monica, surrounded by posters for “Blade Runner” and “2001: A Space Odyssey,” and showed off the results with a scene from “Fall” in which one of the main characters complains about their situation.
“Now we’re stuck on this stupid freaking tower in the middle of freaking nowhere!” Virginia Gardner yelled to Grace Caroline Currey as the two of them huddled together on a platform that looked dangerously high.
After a moment, Mann played back the scene. But Gardner’s words were much harsher this time: “Now we’re stuck on this stupid f****ing tower in the middle of f****ing nowhere.”
The first version was shown in more than 1,500 American theatres in August. But Mann actually shot the second one—the one with dialogue fit for a sailor—back on that fire ant-filled mountaintop. If you didn’t know that a neural network had rebuilt the actors’ faces, you probably wouldn’t know that their cleaned-up dialogue was added at the last minute.
Mann said, “The whole point is that you can’t tell what’s real and what’s not.”
The morals of man-made things
When it comes to making movies, that kind of realism is clear to see as a good thing. No one wants to buy something that looks like it was made in Microsoft Paint.
But the rise of software that can change what someone seems to have said without anyone noticing has big implications for a media world that is already full of fake news. After all, Flawless’s main product is just a more real version of “deep-fakes,” or CGI that makes someone’s face and voice look and sound like theirs.
It’s not hard to think of a troll who, instead of using these tools to cut swear words out of a movie, makes a video of Joe Biden declaring war on Russia and sends it around the internet. Digital porn that looks like someone has also become a problem.
Flawless isn’t the only company that does business in this area, either. The CEO of Papercup, Jesse Shemen, told The Times that the company’s goal is “to make any video watchable in any language.” Papercup makes fake human voices that are used in dubs and voice-overs.
And a mainstay of visual effects Hanno Basse, the company’s chief technology officer, said that machine learning is used to make actors look like they are in scenes where a stunt double is needed.
As these and other companies automate the entertainment industry more and more, there are a lot of ethical questions come up.
Hollywood is already using its new ability to digitally re-create dead actors, like Anthony Bourdain’s voice in the documentary “Roadrunner” or Peter Cushing and Carrie Fisher in recent “Star Wars” sequels. Now, celebrities who have died can also come back to life in hologram form.
Scott Stroud, who runs the media ethics program at the University of Texas at Austin, said that digitally changing dialogue could make it hard for the original people to give their permission. “What the actors thought they were agreeing to isn’t exactly what is made.”
Denver D’Rozario, a marketing professor at Howard University who has studied the software resurrection of dead actors, says that this technology could make it possible to change movies long after they come out.
“Let’s say … “A guy drinks a can of Pepsi in a movie, and 20 years from now, Coke gives you a sponsorship,” D’Rozario said. “Do you switch the Pepsi can to a Coke?” “How long can things stay the same? When can things be purchased?”
Mann said that his technology has many benefits, such as breaking down language barriers and making people more empathetic across borders. It also saves actors the trouble of having to reshoot scenes. He thinks that ideas like D’Rozario getting a Coke sponsorship are new ways to make money.
Mann also said that Flawless has been proactive about making a product that helps real human performance rather than taking its place of it.
“There is a way to use technologies in the same way that the visual effects industry has already set it up, which is to do it securely, legally, and with everyone’s permission,” he said.
The director said that the company has already talked to “all the big unions” about how to make and use this technology in a smart way.
Representatives from SAG-AFTRA emphasized that AI filmmaking technology can help or hurt actors, depending on how it is used.
Jeffrey Bennett, SAG-general AFTRA’s counsel, said in an email, “Technologies that don’t do much more than digitally improve our members’ work may just need the ability to give informed consent and, possibly, more pay.” “On the other end of the spectrum are the technologies that could replace traditional performance or that take our members’ performances and make completely new ones. For these, we insist that they must be a subject of bargaining.”
It’s a train that has already left the station, for better or worse.
“Fall” is currently streaming, and Mann said that other movies that his company worked on will be released this Christmas, but he can’t name them yet.
During the holidays, if you go to a movie, an A.I. might have helped make it.
Can you figure it out? Does it even matter?
