From faces to voices, AIs are conquering cinema

In season 4 of Stranger Thingsin the first of Wednesdayin Watchmen And Umbrella Academy and also in movies like Spiderman: No Way Home And Ms Marvel. In addition to dozens of productions by more or less all studios, from Disney to Warner Brosvia Amazon, Apple, HBO, Marvel, Netflix and Paramount.

L’use of artificial intelligence in Hollywood is not future: it is reality, it has been going on for some time and above all it goes on without the spectators practically noticing it. Which is what should happen with good special effects. It has been going on for some time and is declined along two main lines: the images and the voice, as they explained to us Matt Panousischief operating officer of Marz, e Scott Mannco-founder and current CEO of Flawless.

Grow old (and rejuvenate) with a couple of clicks

Born in 2018, Marz deals with the deaging/aging of actors, i.e. the rejuvenation or aging of their faces, through her Vanity AI: “We use a general database of faces, refined for specific needs, that is, with the data of the actor or actress we need to work on,” Panousis explained to us.

Marz, who has 270 employees in 26 countrieshas invested millions of dollars to get started, above all to buy computers and graphics cards, and does everything independently: “You choose the person on whom to intervene, select the part of the face that interests you and set the parameters to decide the extent of the modification, which may be applied in each scene in which that specific part of the face appears”.

The point to understand is that not only AI manages to rejuvenate or age a facebut who manages to do it at an unthinkable speed: “With traditional methods, to intervene on 5 seconds of footage needed 2 days of work, because the points of the face must be mapped, it must be taken into account that they move if the actor laughs, cries or changes expression, then everything must be harmonized – Panousis reminded us – With our instrument, you can do it all in minutes”. And with much less money.

The next level of voice acting

Speed ​​and economic convenience are the two advantages that Flawless also have: founded in 2018, it has 50 employees and its own AI, called TrueSyncwhich is used for voiceovers and lip-sync.

To understand what he is capable of doing, Mann gave us the example of Bankruptcyone of his films from 2022: “There were too many swear words, and in the US it would have been forbidden for children under 17. To get down to PG13, 35 scenes would have had to be reshot, which is an expensive thing and which cannot always be done, because perhaps the actors are busy elsewhere”. And so “we have used TrueSync to correct 35 words and re-sync lip movement of the actresses as they pronounce them”. How long did it take? “One day to figure out all the words and another to train the AI, who then got to work and finished everything in two weeks.” Objective achieved and “at a tenth of the cost”.

Like Visual AI, TrueSync also starts from a initial database that is adapted according to needs: “The movement of the lips affects other parts of the face, because when you speak, even the eyebrows can move – Mann reminded us – Only an AI can take all of this into account and do it well and quickly”. That’s not all: “Our AI can also translate an actor’s speech into another language and synchronize the movement of the lips to match the new words”. The actor performs in his own language and TrueSync adapts facial expressions to those needed in the various countries, so that the film seems to have been shot (for example) in Italian, Spanish, Japanese and so on.

New version

GPT-4, the artificial intelligence of OpenAI becomes even more powerful

by Bruno Ruffilli

March 14, 2023

Why Hollywood needs AI

The point of everything is to give the viewer a realistic and believable product, and we are indeed we spectators are the main culprits of this revolution: “People ask for quality and they ask for it everywhere, even on TV – Panousis reminded us – Only that quality costs money”. How much does it cost? “You can also spend 5-10 thousand dollars for each single scene with special effects, which can also cover 20-30% of the entire budget of a film – Marz told us again – With us it costs about 50% less and you know immediately how much you spend, because we have a price list that applies to everyone and makes this market more accessible”.

Both companies are aware of the risks that exist for employment, and both work with the unions to reduce them: “The human professionalism remain unsurpassed and are fundamental, but we must protect people’s rights and protect this art – Mann told us – We want those involved in the change to do it in a willing way and to be paid”. According to Panousis, however, “there will be many independent filmmakers that thanks to these tools they will become capable of doing incredible things: we are democratizing creativity”.

@capoema