How we created Google Meet's adaptive audio features

 

Adaptive audio

Adaptive audio: what is it?

A function called "Adaptive Audio" automatically modifies the transparency and noise cancellation settings according to your environment. When necessary, it allows in crucial noises like conversations or alarms while also helping to block out noise in noisy areas. In this manner, you may maintain awareness of your surroundings without manually adjusting settings.

At this year's Cloud Next, the Google Meet team set up a demonstration room with numerous laptops stacked side by side. The team questioned the clients what they would anticipate would happen if every laptop entered the conference room at once after bringing them into the room.

The team started research on adaptive audio after the pandemic caused a global shift to video conferencing and then hybrid work. It was challenging to get new conference room hardware at the time because of supply chain issues. Huib also points out that a lot of companies either didn't have enough video conferencing rooms at first or didn't have the money for specialized meeting room equipment.

Teams wanted to be able to set up ad hoc meeting spaces without being forced to huddle around a single laptop. Allowing everyone to connect from their own devices while keeping the "screams" quiet is much harder than it looks.

Think about a movie theater's sound system. "You have a number of speakers around you, and it's a pleasant audio experience because they're all connected to the same sound source, so they play out in an intended synchronicity," says Meet Software Engineer Manager Henrik Lundin. Now, if several devices were playing the same music in the same room without synchronization, it would sound terrible. You're getting many copies of the same sounds, like you're in a large cathedral. The same is true when you speak in front of multiple microphones on different devices; even though they are not all on at the same time, they all record sound simultaneously.

Next is the echo problem. You've probably noticed that sometimes you hear your own voice echoing when you utilize video conferencing tools."You don't always get that because the devices that conduct meetings have an echo canceller built in," Henrik explains. It's a signal processing technique that aims to distinguish between the parts of the microphone signal that are your speech and the parts that are just coming from the speakers on the device. This becomes 10 times more challenging when multiple laptops are in the same room, connecting to each other's microphones and playing the same music.

To solve this audio problem, the team had to spend a lot of time in the same room and figure out how to have their laptops detect each other as being next to each other. During the conference, they first tried letting participants join groups that had already been formed. Henrik continues, "We were able to test the experience of synchronizing the microphones and speakers on all of the laptops, even though this was obviously prone to mistakes."

After that, they tried employing ultrasonography. The laptops may begin to behave collectively by emitting high-frequency sounds that are inaudible to the human ear when they detect the presence of other computers in the vicinity. This eliminated the need for users to manually configure their devices or select the room they were in. However, as Henrik adds, "it was rather challenging because the ultrasound had to be accurate and reliable on any device if audio leaks from the room next door, it shouldn't think you're in the same room." The researchers employed a new type of ultrasound and changed the frequency and amplitude to increase reach without being audible in order to increase accuracy.

When Meet detects that multiple laptops are present, adaptive audio kicks in right away, syncing the microphones and speakers on each laptop without turning any of them off. It switches between microphones based on who is speaking in order to prevent feedback and echo. Meet also uses a cloud denoiser and backend processing to reduce background noise and enhance audio quality before forwarding Adaptive audio to other participants.

Many Google meetings now use adaptive audio on a daily basis, frequently without the participants' awareness. It is among the technologies that lessen the cognitive load on the user. Ahmed Aly, Meet Interaction Design Lead, says individuals don't have to wonder if they're set up right before attending a meeting. From the end user's point of view, the technology just functions whenever they open their laptop and go to a meeting, regardless of how amazing and complicated it is.


The team continues to research ways to help people interact in the future, especially when there are no meeting rooms or conferencing tools accessible. We hope it enhances meeting fairness and participation and provides greater flexibility," Huib said. "The camera and microphone are right in front of you, so you can be seen and heard clearly from wherever you are sitting."