Sony Shows How Technology Helps Creativity at CES 2020 with Its Virtual Film Set

When Masaru Ibuka and Akio Morita started an electronics company in Tokyo in 1946, they probably didn’t envision that it would become one of the largest corporations on the planet. Originally known as Tokyo Tsushin Kogyo, the company that specialized in tape recorders and radios eventually renamed itself Sony in the ‘50s, because it was easier for Western consumers to pronounce. Sony grew exponentially due to the transistor radio, becoming one of the best-known companies in all of consumer electronics throughout the ‘60s and ‘70s. In the ‘80s it spread into media, buying record companies and movie studios, and in the ‘90s dove into gaming with the PlayStation. By the 2000s Ibuka and Morita’s small concern had expanded its reach into almost all aspects of entertainment, creating both the media that we enjoy and the devices that let us consume it.

As an omnipresent force in the world of consumer electronics, it only makes sense that Sony has an outsized presence at the annual Consumer Electronics Show. This year the company made its wide-ranging diversification the focus of its CES presentation, launching a new “Boundless by Sony” campaign that highlights how Sony technology has impacted—and united—different corners of the media world. This included a panel discussion between three top creatives from the realms of music, gaming and movies: Mike Larson, the sound engineer for Sony Music’s Pharrell Williams; Cory Barlog, the game designer responsible for the most recent God of War game from Sony Interactive Entertainment’s Santa Monica Studio; and Glenn Gainor, the head of physical production at Screen Gems and the president of Sony’s Innovation Studios, which develops new technology for film production.

Selecting these three specific people accentuated almost all of Sony’s various interests, from TVs, to audio equipment, to videogames, film production and beyond. The panel itself didn’t include any groundbreaking revelations—it was just a pleasant chat about the creative process between three high-level creatives whose different disciplines share a few common elements. What it was, though, was an illuminating adjunct to Sony’s CES booth, which focused heavily on 8K televisions, 360 audio headsets and soundbars, and a virtual film set built in part through volumetric data capture. We had already seen these technologies in action, responding to them as consumers; Larson, Barlog and Gainor provided insight from the creators who actually use and design for this technology.

As the head of Innovation Studios, Gainor was able to discuss perhaps the most impressive tech demo on Sony’s show floor, and one that might most succinctly sum up the image Sony hopes to impart through the Boundless campaign. The virtual film set installed at CES included a full-sized Ecto-1 ambulance from Ghostbusters. It sat parked within the three walls of the film set, a crucial practical effect against the lifelike projection backdrop of the Ghostbusters headquarters.

Here’s how the Innovation Studios’ virtual film set works. Imagine a studio that’s essentially a black box. There’s room between its walls for actors, practical sets, the crew and their equipment. Instead of a fully-built out physical set, though, projections of pre-filmed images create a space that looks realistic on camera. The Sony booth ran a constant demo of the tech, with a camera slowly gliding on a rail around the Ecto-1, as the façade of the Ghostbusters building was projected on the walls around it. The feed was streamed to TVs above the set, which showed how seamless and realistic the process looked. Leaves blew over puddles of water on the asphalt in the pre-filmed background, and on the TV screen it looked like it all occupied the same physical space as the car that sat within the set. It even accounts for the angle of the camera and how the lighting and appearance of the projected backdrops should change in relation to it; guests at the Sony booth could see the background image shift as the camera slowly moved, mimicking how the view of a physical building would change alongside the camera’s movements.

As the head of physical production at Screen Gems, a label of Sony’s film division focused on lower budget movies, a big part of Gainor’s job is managing budgets. “I segued into this whole role and position at Innovation Studios, and creating the studio with my friends, because of the necessity technology is now playing for me to even get a film made in the first place,” Gainor told Paste at CES. “I started to engage technologists in the camera department, and inspire ideas and inspire the way cameras were maybe being built, because I was using them differently. And I was using them differently because at my movie label we don’t necessarily have the same amount of money as other labels might have. So I really was inspired to look towards technology to allow me to make film content for not as much money as a competitor might make it for.”

It’s not just tighter budgets that Gainor has to contend with. The amount of content being produced today is significantly higher than it was just a decade ago, before streaming created an expanded demand for original content. The number of available production facilities or crew hasn’t increased alongside that production, though. As Gainor explained, “We’re globally challenged to find locations, to find space to shoot these films. So I envision a world where virtual sets will enable us to tell more stories and not have to move around as much. That’s the dream for me.”

To meet these goals, Gainor started to explore what technologies his parent company had to offer. “What I noticed, just by hanging around at Sony, is that Sony had a bunch of interesting technology—particularly in their camera sensors,” he said. “And I started learning about time-of-flight, and I was just very blessed to be able to understand key players at Sony. So we came up with this idea that we could be sort of a portal for technology and story because when I [started in the industry] it was pretty simple in a sense—it was a film camera. But as we started getting into the digital revolution, technology started playing a bigger role in how we make content. 15 years ago it was pretty much ‘this is how you make a movie.’ Today the rule book is being written again.

“We’re not traditionalists any more. There’s not a way [to make a movie]. I remember, a long time ago, walking on sets with established actors of cinema, and when they first walked on set, if they didn’t see their key light, they’d call out the gaffer and say ‘where’s my key light?’ and today an actor couldn’t really do that, because there is no key light like we used to have.”

Gainor’s innovative approach to movie-making hasn’t made its big-screen debut yet. So far the TV show Shark Tank uses the technology for its virtual set, and it was also used in a commercial connected to the Men in Black franchise. It’s only a matter of time before this process is used in a movie, though. The goal is to not just use it for Screen Gems movies, or even Sony films, but throughout the industry, as a way to, as Gainor explained it, “collapse geography.” He grew passionate when described the virtual set as a form of “archiving the world”—creating a bank of real world assets that could help filmmakers be “in two or three locations in one day.”

The virtual set created by Gainor’s Sony Innovation Studios is an ideal example of the kind of corporate unity and company-wide technological collaboration that Sony was trying to impart at CES this year. An executive at one of Sony’s many branches had an idea, reached out to creators and technologists throughout the company’s structure, and developed a new process and technology that could materially impact how content is produced. Even if it doesn’t work out as Gainor envisioned, it still provides fascinating insight into how a company with as many tentacles as Sony can facilitate not just technological change but creative change, as well.

Senior editor Garrett Martin writes about videogames, comedy, travel, theme parks, wrestling, and anything else that gets in his way. He’s also on Twitter @grmartin.