Researchers at Carnegie Mellon University have developed a camera system that can seemingly detect sound vibrations with a level of precision that makes it possible to recreate the audio without inference or a microphone. A team from CMU’s School of Computer Science’s Robotics Institute (RI) built the system, which has two cameras and a laser. It can detect “high-speed, low-amplitude surface vibrations” that the human eye can’t see, the university said in a press release.
The system features regular cameras rather than high-speed ones used in previous research, which should lower the cost. “We’ve made the optical microphone much more practical and usable,” Srinivasa Narasimhan, an RI professor and head of the Illumination and Imaging Laboratory, said. “We’ve made the quality better while bringing the cost down.”
An algorithm compares speckle patterns captured by a rolling shutter and a global shutter. It uses the differences between the patterns to calculate the vibrations and recreate the audio. A speckle pattern (which is created by the laser in this case) refers to the behavior of coherent light in space after it’s reflected off of a rough surface. That behavior changes as the surface vibrates. The rolling shutter rapidly scans an image from one end to the other, while a global shutter captures an entire image at the same time.
“This system pushes the boundary of what can be done with computer vision,” assistant professor Matthew O’Toole, a co-author of a paper on the system, said. “This is a new mechanism to capture high speed and tiny vibrations, and presents a new area of research.”
The researchers say they were able to isolate the audio of guitars that were being played simultaneously. They claim that the system was able to observe a bag of chips, and use vibrations from that to reconstruct audio being emitted by a nearby speaker with higher fidelity than previous optical microphone approaches.
There are a lot of potential applications for this tech. The researchers suggest, for instance, that the system could monitor vibrations from machines in a factory to look for signs of problems. Sound engineers could also isolate the sound from an instrument to improve the mix. In essence, it could help eliminate ambient noise from audio recordings.