Deepfake AI Hijacks Live Conversations in New Audio Attack

IBM Security researchers have discovered a concerning technique known as “audio-jacking” that allows hackers to hijack and manipulate live conversations using artificial intelligence (AI). This attack method relies on generative AI and deepfake audio technology. During their experiment, the researchers instructed the AI to process audio from two sources in a live communication, such as a phone conversation. Once the AI detected a specific keyword or phrase, it intercepted the related audio, manipulated it, and then sent it on to the intended recipient. The experiment was successful in intercepting a speaker’s audio when prompted to provide their bank account information, and the AI replaced the authentic voice with deepfake audio containing a different account number. The victims were completely unaware of the attack.

The blog post from IBM Security emphasizes that while executing this attack would require some level of social engineering or phishing, developing the AI system itself was surprisingly easy. The researchers spent most of their time figuring out how to capture audio from the microphone and feed it to the generative AI. In the past, creating a system to autonomously intercept specific audio strings and replace them would have required a complex computer science effort. Modern generative AI technology simplified this process. The blog post notes that only three seconds of an individual’s voice are needed to clone it, and these deepfake technologies can now be accessed through APIs.

The threat of audio-jacking extends beyond tricking victims into depositing funds into the wrong account. The researchers highlight that it could also be used as an invisible form of censorship, allowing hackers to change the content of live news broadcasts or political speeches in real-time. This potential misuse poses a significant concern for the security and authenticity of audio communication. It is important for individuals and organizations to be aware of these vulnerabilities and take necessary precautions to protect themselves from such attacks.

The researchers’ experiment serves as a wake-up call for the potential dangers posed by advances in AI and deepfake technology. As these technologies continue to evolve, it becomes crucial to enhance security measures and develop effective countermeasures to mitigate the risks. To address this growing threat, a collaborative effort involving experts from various fields, including computer science, cybersecurity, and psychology, will be necessary. By understanding and staying ahead of these emerging risks, we can work towards a safer and more secure digital environment.