Where are you using WebAssembly?
Wasm promises to let developers build once and run anywhere. Are you using it yet?
At work, for production apps
At work, but not for production apps
I don’t use WebAssembly but expect to when the technology matures
I have no plans to use WebAssembly
No plans and I get mad whenever I see the buzzword
Open Source / Software Development

Pixar CTO on OpenUSD, Vision Pro Future, and USD Web Tools

We spoke to Pixar CTO Steve May about new developments in USD, an open standard for 3D content being used by Apple, Meta, Nvidia, and others.
Feb 8th, 2024 9:00am by
Featued image for: Pixar CTO on OpenUSD, Vision Pro Future, and USD Web Tools
A USD view of the Pixar movie, ‘Coco’; via Pixar

Pixar open sourced its 3D graphics technology Universal Scene Description (USD) back in 2016, a few years after it was created for its own movie-making purposes. USD has since been adopted by industry heavyweights like Apple, Meta and Nvidia. Some have likened USD to the Photoshop format, psd. The hope is that it will eventually become the default language for a 3D internet — perhaps akin to what HTML is for the World Wide Web.

To find out how USD is being used today and how it may further evolve in an era where “metaverse” and “spatial computing” are major buzzwords, I spoke to Steve May, Pixar Chief Technology Officer and also chairman of the non-profit Alliance for OpenUSD (AOUSD).

What Is USD and How Is It Used?

In a profile of May released today by the AOUSD, USD is defined as “a universal language for all elements in a digital scene — characters, environments, lighting, and camera movements.”

When I asked May about how USD is used within Pixar, he acknowledged that it’s a complex technology — but also very powerful and flexible.

“We don’t put all the data into one single entity, one single asset,” he explained. “It actually has an entire web of assets. So you might have an asset that represents the entire scene from one of our films, but then it has these tendrils — these pointers out to other assets, like an asset for each character, and an asset for the set pieces, and an asset for each prop that’s in the set.”

So when a digital artist or developer is working with USD, they’re able to work on a specific piece of 3D content and then connect it with other assets.

Pixar production process

Pixar production process; image by Pixar.

“And then a character has assets itself,” May continued, “because the character may have the body […] but then they may have separate USD files for clothing, for hair, for describing the way that the skin reflects light. All the different parts of it are actually composed of hundreds [of USD files].”

This means that a single scene from a Pixar movie can have thousands, even millions, of separate USD files.

“So one of the real powers of USD,” said May, “is not even just describing say the geometric shape of an object or the material properties of it that drive how it looks, but that it can actually assemble and connect all these assets together in a coherent way, and lets you easily edit individual pieces of it.”

According to May, USD is also well suited to building large virtual worlds or complex 3D industrial applications, because in those kinds of products “a lot of things are happening and changing at the same time,” and require a lot of collaboration.

“It’s much more than just kind of describing stuff,” he said, about the USD format. “It’s also the ability to edit and assemble and manage these really complex networks of assets, in an efficient way.”

USD Beyond Movie-Making: Apple, Meta, and Others

So how has USD evolved since it was open sourced in 2016, and does Pixar still lead the development?

“It has evolved technologically since we open sourced it,” May said (and he confirmed that Pixar still leads the development effort). “It is not a finished technology. So even though it’s fairly mature, and we’ve been using it for a long time — it’s based on literally decades of stuff we’ve done at Pixar — it’s still evolving.”

May noted that USD is “also changing in terms of who’s using it and who’s interested in it.” The initial idea when USD was open sourced was to allow other companies in the film industry to connect with Pixar’s pipeline. Or as May put it, USD was open sourced “so we could access new tools and they would speak USD and then they would just work with everything else — all our proprietary tools that we have at Pixar.”

However, what ended up happening was that other industries wanted to adopt USD as well — everything from Nvidia’s industrial 3D products, to Apple’s mixed-reality products, to Meta’s grand vision to build a 3D “metaverse.”

“A little bit of a surprise was how fast some companies outside of film became interested,” said May. “And two of the early ones, and biggest proponents that are also founding members of the Alliance for OpenUSD, are Apple and Nvidia. And they got behind it really early on, to solve completely different problems.”

These different companies came together in AOUSD, a non-profit organization affiliated with The Linux Foundation. Adobe and AutoDesk were the other founding members, but the organization also now includes Meta, Epic Games, Ikea, and multiple other companies interested in building out the 3D internet.

Vision Pro and Potentially Using Pixar 3D Content

The biggest news in the world of 3D content this year has, of course, been the release of Apple’s Vision Pro headset. I asked if May had tried it out yet. He replied that he has only played with it a bit, but he’s already impressed by its potential.

“I think it has aspects to it that are really kind of mind-blowing,” he said. “And I think everyone knows, it’s just the beginning also. And so for me, as much as anything it gives you a window into the future. It kind of jars your brain to think about how you would actually work differently and think about computation differently.”

He added that Vision Pro potentially offers more than previous VR devices that took an “entertainment as a gaming kind of device” approach.

I asked if Pixar is planning to do any applications for Vision Pro, and maybe even on Meta’s 3D internet platform in the future.

“So we are a studio [and] we are very, very focused on making movies — like, that’s what we do,” he replied. “We make movies and then we’re part of a much bigger company, Disney, that helps us distribute those and figure out how to leverage those movies to make appealing attractions at theme parks, appealing consumer products, and things like that. So we’re not specifically tasked with making interactive AR or VR content yet, but we do use those tools to make our movies — so, for example, we will use VR to do location scouts.”

However, he did acknowledge that Pixar’s parent company Disney is well placed to do 3D content for devices like the Vision Pro. He pointed out that Disney CEO Bob Iger was at the Vision Pro launch event, showcasing Disney interactive content.

“There’s just a ton of potential, I think, to find new ways to tell stories and to allow audiences to experience Pixar characters and Pixar movies in new ways,” May said.

USD and the Web

In December, the AOUSD laid out its roadmap for finalizing the specification. Much of the focus is on defining the core spec, but there are other areas being explored too. One I specifically asked about is web support.

“One of the ongoing efforts is to really make it [USD] suitable for web use,” said May. “There are examples, there are prototypes that exist. In fact, if you go to Apple’s product website and you look at the Apple Watch, they have a thing where you can look at it in AR, from your website. That is actually a USD asset that you can view on your phone. So there are capabilities that are there, but there’s much more work to do to really enable it for the web. But that’s what a lot of us in the Alliance are interested in supporting.”

Related, there’s also an ongoing effort to make USD authoring a simpler experience — just as HTML editors emerged in the 1990s to make writing HTML easier.

“You can actually type USD into a text editor, but we don’t really recommend it, because it’s fairly complex,” said May. “So it’s generally done through programming API’s. And I don’t have any reason to think that we won’t have good tools for creating those assets and making them available on the web. Our goal [is] that the same USD asset or assets — whether it’s on the web, or on a device like Apple’s Vision Pro, or it’s on a desktop computer or workstation — the same asset should be ingestible on any of those different platforms.”

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.