How Fran works, the AI ​​that rejuvenated Harrison Ford for Indiana Jones 5

Harrison Ford is about to turn 81but in the next film by Indiana Jones he will have 35. For 25 minutes. He will be 35 for the first 25 minutes of the filmbut not thanks to an actor who plays him as a young man, as happened in Indiana Jones and the Last Crusade with the late River Phoenix: it will happen thanks to artificial intelligence.

What we will see in Indiana Jones and the Dial of Destiny (which is the real title of the film that everyone knows as Indiana Jones 5, expected in theaters on June 30), is only the most recent exampleand perhaps the most sensational and important, of a phenomenon that has been going on for at least 3-4 years in the world of cinema: the rejuvenation, or aging, of the faces of actors and actresses with the use of AI.

In English it is called deaging/aging, we have often written about it on Italian Tech since the beginning of 2023 (last time here) and the difference with what happened (for example) in The Irishman is that it is precisely done not through special effects as we have understood them up to now, using multiple cameras, infrared and computer graphics, but leaving all the work to an artificial intelligence.

youtube: latest trailer for Indiana Jones and the Dial of Fate

How deaging is done with AI

Last March, Matt Panousis, chief operating officer of Marchone of the most active companies in this field, told us that their Vanity AI uses a general database of faces (like this), refined for the specific needs of a specific production, i.e. with the data of the actor or actress on which one must intervene. The data and above all the photographs, so that the AI ​​can imagine how that looks it could change with 5, 10, 15 years less or more.

Subsequently, with the help of quite powerful graphics cards and after having selected the part of the face of interest and having set the parameters to decide the extent of the modification (which will be applied in each scene in which that specific part of the face appears), the artificial intelligence is allowed to process everything and return the scene with the necessary modifications: “With the methods traditional – Panousis had explained to us – to intervene on 5 seconds of footage it took 2 days of work, because the points of the face must be mapped, it must be taken into account that they move if the actor laughs, cries or changes his expression and then harmony must be given to everything. With our tool, you can do it all in minutes.”

youtube: how Fran works

Fran, the AI ​​that learned from another AI

What will happen in Indiana Jones and with Harrison Ford’s face is more or less similar, but with a couple of differences significant.

There Before it’s that artificial intelligence that does the job, that’s called Fran (the acronym stands for Face re-aging networkwhich in Italian is more or less Network for the re-dating of faces), does not belong to some small startup but has been developed internally by Disney, which announced it in late 2022. It is an important point, because it confirms how essential these technologies are becoming even for the biggest studios, which obviously try to bring them into their homes also for economic reasons.

There second diversity is that Fran has not been trained on the faces of real people, captured (as everyone does and it is quite legal to do) through the scraping of the Internet and social networks: USA the so-called synthetic data (what are they?). And this is both a good thing and a useful thing: it’s good because it’s a more privacy-friendly practice and it’s useful because, as the developers themselves explained, it would have been “practically impossible” to train it in any other way to get the desired results. Why? Why would it have been necessary to have “pictures in pairs showing the same subject with the same facial expression, the same pose, the same lighting and the same background in two known and different moments of his life”. In two different ages, in short.

Hence the idea of ​​using synthetic data: Fran’s developers created a database consisting of several thousand randomly generated faces, faces of non-existent people probably created with tools like Dall-E 2, Midjourney and Stable Diffusion. Then they aged or rejuvenated those faces using some machine learning tools and then fed all the data into Fran so she could do the same things and predict which parts of a face would be affected by aging (or from rejuvenation), where wrinkles would appear, where skin would be smoother, and so on.

Yes: it’s somehow an AI that learned to do things from another AI and maybe that’s a little worrying. But Harrison Ford cares little about that, at least for now.

@capoema