Immersive media in the new roaring twenties
Real-time game engines are enabling more people to build immersive stories, vibrant worlds, and engaging characters.
Why are people so excited about spatial computing? There’s a lot of hype about AR/VR/MR/etc. right now, but what’s actually happening? Is this going to be a thing? What will that thing look like? Rather than getting (too far) into speculation and vision of what’s coming, I will go deeper on what’s actually happening, right now, focusing on the tools and trends that are enabling the creation of immersive experiences and communities. Games are the new video. A quick high-level summary before diving into the details:
- Unity and Unreal Engine power thousands of games on all platforms, freeing developers to focus on design, story, gameplay. They continue to push the state of the art and are crossing the uncanny valley.
- More games are including camera modes (RDR2, Spiderman, Last of Us 2, AC Discovery, CP77) and full-on creative versions (fortnite, roblox, dreams, Mario maker)
- Huge communities of designers and developers are flourishing powered by dozens of tools
- We’re just getting started.
So how do we know this is actually a big deal, and will be a major shift in technology and culture? There is plenty of evidence from all the biggest brands and companies, who are investing heavily in various pieces of this future. They clearly don’t think this is a passing fad. Here are some examples, but this is by no means exhaustive:
Apple’s ARKit for SLAM position and motion capture, lidar sensors on new devices, the Metal graphics engine, and the smartglasses product that they haven’t launched but everyone knows is coming. And with their USDz file format, they are even somewhat supportive of WebXR and using the browser for this content, not just native application frameworks.
Microsoft sees Mixed Reality as one of three pillars of their future growth alongside artificial intelligence and quantum computing, with hardware like Hololens 2 and Azure Kinect, as well as gaming properties like Minecraft and Altspace, and of course Xbox.
While they tend to kill almost as many projects as they launch, Google has a number of related projects. They just open sourced TiltBrush and have ended a lot of VR-focused products, seemingly giving up on spatial computing. I think they are taking a similar approach to my thinking in this space: they started with VR, but realized there’s a lot more going on and VR isn’t enough. They are still working on Stadia for streaming games (though they have just killed their internal game studio before launching anything), and they still have some very interesting AR products like Lens, Maps, and 3D models in search results.
Facebook has almost too many to mention; Zuck clearly sees this as the future and a critical market for them to own. They ‘missed’ becoming the mobile platform, and are fighting insanely hard to win the next platform. Oculus is the most obvious product, but there is also the 3D art app Quill, their forthcoming VR-first social network Horizon, and all the AR features on Instagram and the FB app. They’ve also announced smartglasses that should be unveiled soon.
Disney absolutely understands the significance and opportunity, and I would not be surprised in the least if they acquired Epic in 2021. Ignoring all the games they released over the years, they have increased their explorations into VR storytelling, they have a strong partnership with Epic to include a lot of their IP in Fortnite, and have been using new virtual production techniques with Unreal Engine to produce the Mandalorian and much more. They have also started to talk about a “theme park metaverse”.
Snap is likely doing the best job with AR products currently with the innovations around face filters, landmarkers, and their Spectacles hardware product. They are even building in machine learning with SnapML.
Despite recent layoffs, Mozilla has put out some amazing work for spatial computing on the web, most importantly Hubs, which also includes a great editing tool called Spoke. They killed their Reality browser for VR, however.
Lego has been big on AR since it came to iPhones, with AR games connected to new sets, and they just teased an AR video app called Vidiyo. They know how compelling this stuff is for their audience.
Finally, there are many fashion brands, pop stars, and other huge brands investing and experimenting in these spaces.
But there is another company that has been amassing the building blocks of this potential future, and is leading the conversation around the opportunity of the metaverse: Epic (not trying to get into the debate about the definition or feasibility of the term ‘metaverse’, but Epic’s CEO Tim Sweeney has been very outspoken about his ambitions in building it).
Epic’s Unreal Engine provides the real-time rendering engine as well as high-quality assets, character creation tools, story and camera controls, distribution and scale tools, across all platforms. Most of this they provide for free as well, at least as certain volumes of usage. They have been gradually amassing the building blocks of virtual life:
- They acquired Twinmotion and integrated it into Unreal, providing architectural modelling tools to create realistic environments with natural lighting.
- They own Quixel, which offers a massive library of photo-realistic textures and assets to fill out virtual worlds.
- 3lateral and Cubic Motion were combined in Unreal to bridge the uncanny valley and make character models that are indistinguishable from human.
- Beyond these tools for making more immersive and realistic virtual worlds, they are also making it easier to build entire businesses with Epic Online Services, which offers game components like community, cloud syncing, matchmaking, and more to developers.
This is all built in to a real-time engine, avoiding extra complexity and cost around rendering and lighting and more. So users don’t have to settle for static cut-scenes, but can experience the full environment.
Additionally, the next generation of the engine, Unreal Engine 5, will be available this year, which will provide even more benefits automatically. In addition to performance and quality improvements, UE5 includes systems to automatically optimize applications to scale up or down the quality depending on the hardware. Unreal is already in use across all platforms (iOS, Playstation/Xbox, Switch, PC, WebGL, etc), and these enhancements will make it even better no matter where you’re playing.
Finally, they are becoming the defacto standard for virtual production, powering most of the Mandalorian and transforming moviemaking.
For a detailed analysis of everything Epic is doing and what it means, this essay series by Matthew Ball is a must-read.
Looking beyond how major companies are investing in spatial computing, there has been a ton of innovation in 3D creative tools, for professionals as well as amateurs, and even for gamers. These tools are getting simpler and more intuitive so they are more accessible that anyone can create, share, and remix at any experience or quality level.
There are many different professional tools across the pipeline for modelling, rigging, etc that are more people are learning, but these tools are also heading towards simplicity and standardization. It doesn’t matter which tools you choose when the output can be imported to any other system.
The meta effect of the 3D graphics tooling getting productized and simplified is that the tools of game creation are being baked into the games themselves, turning the players into creators. From character creators in games to photography modes to especially creator modes and sandboxes, new generations are growing up with creating and remixing as standard, expected behavior. I grew up with graphic (and then web) design, my kids are growing up with level builders and Tilt Brush.
Back to Epic for a moment: it’s worth mentioning here that they have been using Fortnite to test out these new tools and experiment with game mechanics, and then bundling these new features into Unreal for anyone to leverage.
With all these tools increasing access to 3D design, all these creators are coming together online to share their creations, and their virtual lives. And these new generations expect to join communities around these activities.
Sketchfab and ArtStation are huge communities of digital artists already using all these tools and building business and communities around their work. And the games with creative modes mentioned earlier of course all have active communities sharing their unique experiences: Hogwarts Castle built in Fortnite Creative, Kings Landing recreated in Roblox, impossible Mario Maker levels, Mass Effect or Fallout 4 recreated in Dreams, among endless examples.
The really powerful thing about these budding marketplaces, though, is that they provide effectively unlimited new content for end users who know nothing about these new tools. You don’t have to get into the creative tools to benefit from them. Roblox isn’t popular because of levels included with the game, but the thousands of unique experiences built by a tiny fraction of the community.
Another critical tool that is enabling millions of creators everywhere is of course the smartphone. Software advances focused on augmented reality as well as better camera hardware, including new sensors like Lidar, make it easier and more fun and create new categories and business models.
Built-in depth cameras make it possible to digitize and share real-world objects and experiences in addition to putting digital experiences on top of the real world, which will accelerate the shift towards powerful yet simple creative tools. Today’s iPhone can create rigged 3D models of people and places/things, record motion capture of human movements, and become a virtual camera for digital experiences. The quality is not nearly the same as the professional tools, but remember the image quality of early iPhone photos? And which will improve faster, the hardware components in the devices, or the software (supercharged with machine learning) to process the images?
AR filters are getting more aware of the world they are placed in, and more elaborate as creators keep raising the bars. The next step, though, is mixed reality capture — where a real-world user is filmed playing a VR game and green-screened into the game itself — linking a phone camera up with gameplay. These depth cameras create a virtual green-screen, removing everything behind the player automatically.
When Apple unveiled their new AR-focused 3D file format, USDz, they updated iOS and their apps to natively read these files and view them in AR. The most significant app to support these files is Safari. Now there would be native 3D file viewing built-in to the default browser on billions of devices. Sure there are loads of apps out there now doing interesting things with 3D content, but they are all constrained by the app marketplace. Apps only work if you (or your entire network) install them on your phone. The simplicity of sharing a link and everything working is just as magical in 2021 as it was in 1991. Don’t call it a comeback.
Developers have been able to do a lot with only what the popular browsers support and not dependant on SDKs or app installs or restricted by store rules. Web-based tools have caught up with the native SDKs, and the experiences created there can go viral much easier. And progressive web apps eliminate the visual/functional distinctions between websites and apps even further with notifications, local storage, and cleaner UIs.
As the major players in gaming experiment with streaming games and get blocked from Apple’s App Store, they have been forced to rediscover the magic of the web. There are different approaches to this, like streaming the entire game via Google Stadia, Xbox Game Pass Ultimate, nVidia Geforce Now, or Amazon Luna, or with advances in WebGL bringing desktop-class graphics onto the web. This comes full circle as some of the 3D design tools are browser-based. Figma is the most popular example, but Rive and Spline are similar powerful graphics tools.
As people buy basically everything online, especially during the pandemic, they might have the native apps for their favourite brands/stores, but there are plenty of transactions that don’t justify downloading an app. Particularly on iOS with ApplePay built into Safari. Users are perfectly happy with paying real money to websites over apps.
The final, and most important but also most nebulous, component for building the future of entertainment is artificial intelligence. It has been and will continue to change everything. Machine learning models generally have two phases: learning and generating. You need to train your ML system before it can provide any value.
There are many cases where an ML model could do amazing things, but getting it properly trained is untenable. Some of the most advanced AI systems today, meanwhile, have been trained on video games and in simulations. Computer vision models for estimating human poses, for example, can be improved with training data that includes dozens of camera angles and dozens of different bodies making similar poses — there could be orders of magnitude more training data for the model than was cost effective previously.
From there, machine learning can accelerate unique content creation in numerous ways. It will fill in gaps in 3D scans, automatically rig models, improve the resolution and detail, smooth out lighting or movement errors, and supercharge the workflows around creating and customizing 3D assets. It can compose effectively infinite variations based on human-created content, and when combined with user behavior and interest data it can optimize for what people like and want.
So what does all of this mean for content platforms looking to stay relevant in the coming years? Consumers and creators will use simple tools like 3D scanning and motion capture on their phones to create animated 3D models based on the world around them. AI-powered software will upscale and improve the quality and variety of those scans, make remixing them easy, and automatically import them into photo-realistic real-time game engines. An open marketplace of these models and animations will allow anyone to build an audience and generate revenue off these creations. At least until the AI can create everything for us, automatically.
