The technologies that will power the metaverse (one day)

With all the talk of the metaverse over the past few months, you might think it’s already open for business. It seems that every day I’m fed an article through my Google News feed, my Twitter feed, or even just when I’m browsing Reddit. Despite all the hype around it, though, many people still aren’t clear on what the metaverse actually is. 

Despite Facebook’s rebranding itself as Meta, the company isn’t the one that came up with this idea. The term actually comes from Neal Stephenson’s 1992 science fiction novel “Snow Crash,” and the idea of a metaverse has been at the center of many other science fiction stories, such as “Ready Player One.”

“I think that in general the metaverse is still a very early-stage concept. Even some of the large companies demonstrating Metaverse experience, to me, I feel it’s very rough today,” said Tony  Zhao, CEO and co-founder of Agora, a provider of conversation technologies. “It’s not even close to what some of the movies are trying to present, like ‘Ready Player One.’ If you watch that movie, it’s more like another life, where you’re living in a totally virtual environment, but feeling like reality. This technology is still very far away from that.” 

Obviously we’re not quite at the level of technology featured in those worlds, so what does a metaverse look like in our world? 

According to Gartner, a metaverse is “a collective virtual open space, created by the convergence of virtually enhanced physical and digital reality. It is physically persistent and provides enhanced immersive experiences.”

On one extreme of this is the idea of putting on VR headsets to virtually join a work meeting or event, while at another end of the spectrum, people are already participating in a metaverse of sorts when they log in to a MMORPG like “World of Warcraft.” Current implementations of the technology result in several tiny metaverses, but Gartner envisions them all converging. 

“Vendors are already building ways for users to replicate their lives in digital worlds,” said Marty Resnick, research vice president at Gartner. “From attending virtual classrooms to buying digital land and constructing virtual homes, these activities are currently being conducted in separate environments. Eventually, they will take place in a single environment – the metaverse – with multiple destinations across technologies and experiences.”

Gartner predicts that by 2026, a quarter of people will spend at least one hour of their day in a metaverse for work, shopping, education, social media and/or entertainment purposes. 

According to Zhao, there are a number of efforts happening right now to create an environment for people working together in distributed environments to feel like they are working alongside coworkers. In education there are also efforts being made to create study rooms where students can virtually get together and study with peers. 

Learn, explore, prepare

This might be exciting for some to think about, but it’s important to stay grounded and keep in mind that the metaverse is still in early stages and will require a number of different technologies in order to function. Gartner says that since the metaverse is still at an early stage, companies should limit their investments in it at the moment, and instead prioritize learning, exploring, and preparing for it. 

“To understand the concepts of a Metaverse, think of it as the next version of the Internet, which started as individual bulletin boards and independent online destinations. Eventually these destinations became sites on a virtual shared space — similar to how a Metaverse will develop,” Gartner wrote in a post.  

Virtual reality is at the heart of many people’s visions for the metaverse, but there are many ways in which virtual reality will need to evolve to support this future.

According to Zhao, one technology that will be crucial for the metaverse is support for real-time engagement. 

“Everybody is talking about the metaverse right now, and there are a lot of different views or opinions around that,” said Zhao. “But one thing we see as a common factor in there is real-time experience, in which people should be able to interact in real time, within that metaverse.”

In order to ensure a smooth experience, it’s important that you’re able to have a stable connection across all clients. The difficulty of this varies if you have four people interacting versus say 4,000, Zhao said. 

“It takes some work in the back end, to make sure it’s both connected, but also, stable and in high quality for real-time traffic of the audio or video data, the virtual environment data,” said Zhao. 

Dealing with lag in normal settings on the internet, whether it be interacting with a website or playing a video game, is already annoying. But when in an immersive environment, a small amount of lag will take you completely out of the experience. 

“And we’re talking about just a few milliseconds,” said Asaf Ezra, co-founder and CEO of Granulate, a workload optimization company. “You don’t have those hundreds of milliseconds that it will take you to go back to the back-end service somewhere far, which 5g is trying to alleviate. And once you’re there, you have your application, you have your computer and you need to return a response. The round-trip time end to end should be just at most, what we understand is around 50 milliseconds for people to see it as immediate. Right now, if for example, we were to send a request to US-East on AWS, just a round-trip time would take about 100 If not 200 milliseconds.”

VR sets need to advance

Visually, the resolution of current VR headsets and programs still has a way to go, according to Ezra. Currently, the resolution offered through a VR headset makes it difficult to read letters in text, he explained.

“So if you expect people to be able to walk down the street and see advertisements on buildings based on wherever they seem to be, or augment data onto their screen … about the weather and stuff, I think we’re very far from that,” said Ezra. 

Current compute levels also can’t support a “Ready Player One”-type metaverse. Ezra gave the example of “World of Warcraft.” Instead of having millions of people playing at once, imagine if there were two billion people. All of a sudden you’d need to be calculating what each player sees on their screen from their perspective, when each of those billions of people are doing different things. 

“So you’re not only talking about a much, much larger amount of compute, but also the transformations that you have to make to see it from everyone’s personal vision, that would take a huge leap forward in what we expect to see,” said Ezra.

The current chip shortage is certainly playing a role in how these compute issues can be resolved. For example, Facebook and Google are two companies currently pouring a lot of money into the metaverse and virtual reality, but that also requires specific hardware, which can be hard to obtain at the moment. Ezra expects that because of this, we’re at least a few years away from the place we need to be. 

Another area Ezra pointed out with the hardware is power consumption of the headsets themselves. This includes both battery life and power to cooling systems.

Considerations for metaverse developers and adopters

The metaverse was a topic in Thoughtworks’ most recent installment of Looking Glass. In the report, the company details some of the things that developers should consider as they begin developing for the metaverse. 

According to Thoughtworks, developing for the metaverse will be different from developing web-based applications, since users will interact with the metaverse in a different way than they’ll interact with web-based applications. For example, the UI will be completely different, and a lot of interactions might be done with hand gestures, rather than tapping on a screen or pointing and clicking with a mouse.

So, to meet users’ needs, developers will have to approach development from a different perspective. 

The different user experience will shape the way that developers approach working on the metaverse. For example, according to Thoughtworks, when developing for metaverse, emotional interactions must be considered as well. “People represent differently in virtual worlds, which can have moral and ethical implications,” Thoughtworks explained in its report. 

They also cautioned developers to prepare for a certain degree of vendor lock-in, but also recommended that they be open to change. “Embracing one platform may be the best solution for your organization now but not necessarily over the longer term, depending on how the ecosystem and your needs develop,” they wrote. 

Another thing to bear in mind is that the rollout of solutions will be constrained by the availability of certain capabilities and technologies.

Finally, the metaverse will present several B2B opportunities for developers. For example, training, conferencing, gaming, and virtual worlds can be explored, but there are also a number of new and inventive ways to use the new technologies, such as intelligent, self-piloted drones in agriculture or rescue use cases. 

As early as next year businesses might “begin to understand that the expanding frontiers of interaction  don’t just pave the way for richer customer experience but can actually drive business and process improvements, by pairing technology-based speed, scale and precision with human capabilities and ingenuity,” said Kate Linton, head of design at Thoughtworks.

What the current state of the metaverse could look like

So given that the “Ready Player One” view of the metaverse is still a long way away, what sort of metaverse can we expect to be able to experience now, or in the near future? Ezra expects that perhaps large events can be experienced in VR with the current technology. 

For example, he can envision people experiencing sporting events or even concerts together in virtual reality. 

Users would be able to switch between different viewpoints of where to view the event from, rather than being constricted to whatever main camera is being broadcast.

Inside the Meta Lab

Facebook, now called Meta, held an online event on Feb. 23 in which Mark Zuckerberg laid out his vision for how AI will be used in building the metaverse. 

During the event, he revealed a number of breakthroughs that the company has made. One is BuilderBot, which is a tool that allows voice commands to be used to generate or import things into a virtual world.

The company is also working on a Universal Speech Translator that will provide speech-to-speech translation across all languages.

Another speech technology that was highlighted is Project CAIRaoke, which is an AI model for talking with virtual assistants that will enable more fluid and natural conversations to take place.

“We can imagine that in a few years, the technology from Project CAIRaoke will underlie next-generation interaction between people and devices. On devices like VR headsets and AR glasses, we expect this type of communication to eventually be the ubiquitous, seamless method for navigation and interaction, much as how touch screens replaced keypads on smartphones. Our current model is an important step forward, but we have more work to do to fully realize this vision. We are excited by both the progress we’ve made so far and the challenges ahead,” Meta wrote in a blog post explaining the technology.  

A new resource is also being made available to help people better understand how AI systems work by outlining the models that comprise an AI system and how they interact. 

The company also is launching an initiative to bring talent from underrepresented groups into AI. Through the AI Learning Alliance, coursework on machine learning topics will be made available. The coursework will be developed by professors at universities with large populations of students from underrepresented groups, Meta explained.

And finally, it is releasing the open source library TorchRec, which is used for building recommendation systems (RecSys) for PyTorch, and is used to provide personalization for several of Meta’s products. 

PyTorch started receiving a lot of feedback in mid-2020 that there wasn’t a large-scale recommender systems package in the PyTorch ecosystem, and as they worked to solve that gap, engineers at Facebook approached the community about contributing its library to PyTorch and building a community around it. 

“This seemed like a good idea that benefits researchers and companies across the RecSys domain. So, starting from Meta’s stack, we began modularizing and designing a fully-scalable codebase that is adaptable for diverse recommendation use-cases. Our goal was to extract the key building blocks from across Meta’s software stack to simultaneously enable creative exploration and scale. After nearly two years, a battery of benchmarks, migrations, and testing across Meta, we’re excited to finally embark on this journey together with the RecSys community,” Donny Greenberg, product manager for PyTorch; Colin Taylor, senior software engineer for Facebook AI; and Dmytro Ivchenko, software engineer at Facebook, wrote in a blog post.