Sora and video AI

OpenAI has released Sora, an artificial intelligence model for generating videos capable of creating realistic footage that adheres to the physical laws of the real world. The videos generated by Sora are striking not only for the quality of the images but also because the model seems capable of “understanding” how to reproduce characters moving and performing actions while respecting the boundaries of the physical world. According to OpenAI, Sora represents a fundamental step in the evolution of generative AI, as it is akin to “teaching AI how to understand and simulate the moving world.” The current version of the model can generate videos from text prompts, similar to ChatGPT and DALL-E 3, producing videos with a maximum duration of one minute that maintain visual fidelity to the user’s input.

The sample videos published by OpenAI are impressive. In one, a woman walks in a city reminiscent of nighttime Tokyo. In another, mammoths run in the snow, overlapping each other without interrupting the image’s continuity. In a third, a dog walks from one windowsill to another, giving the impression of neither floating nor flying but respecting the physical representation of gravity that our brains would expect. What Sora Can Do OpenAI didn’t allow me to submit requests to Sora, but it shared four examples that show the potential of its new AI (but none are close to a minute in length, maxing out at 17 seconds).

The first comes from a detailed prompt that sounds like it was written by an obsessive screenwriter: ‘A magnificent, snow-capped Tokyo is abuzz. The camera moves through the crowded streets of the city, following several people enjoying the snow and shopping at nearby stalls. Gorgeous Sakura petals fly in the wind along with snowflakes.’ The result is a convincing film set in a very recognizable Tokyo, in that magical time of year when snowflakes and cherry blossoms coexist. The virtual camera moves as if attached to a drone, following a slowly strolling couple. One of the passers-by is wearing a mask.

To their left, cars whiz by on a road that runs alongside a river, while to the right, customers enter and exit a series of small shops. The video is not perfect. Watching the video several times, you realize that the protagonists – a couple walking on a snowy sidewalk – would have found themselves faced with a dilemma if the virtual camera had continued to roll. The sidewalk on which the two are walking seems to be a dead end, which would have forced them to climb over a small guardrail to reach a strange pedestrian crossing on their right. Despite this small inconvenience, the Tokyo example represents an extraordinary exercise in world-building.