Google's Viral Gemini Video was Fake

What is more exciting than the AI industry? Likely not a big deal but still interesting to read!

Source: Google ~ Superhuman Newsletter

“Google, this is embarrassing“ tweets Machine Learning engineer Santiago, describing one of Google’s demo videos for its new AI model Gemini which has generated millions of impressions across different social media platforms.

The video in question shows Gemini seamlessly answering questions about several images that are being shown to it. However, there’s one big problem with this video: it’s not happening in real-time like it’s being shown. According to a Bloomberg article, the video demo “wasn’t carried out in real time or in voice.“.

This information has cast some doubt over the model’s features and its performance. Many social media accounts and some media outlets have called the video ‘fake.‘

Another point of debate is how well Gemini performed on the MMLU, a popular benchmark used to evaluate the knowledge and problem-solving ability of AI models.

Google claimed that Gemini was the first AI model to outperform human experts on the test. However, Brett Winton from ArkInvest and others pointed out that the results were achieved by deploying certain prompting techniques, and that Gemini is likely behind both human experts and GPT-4 on the benchmark.

While some of the frustrations and criticisms leveled at Google are understandable, accusing Google of ‘lying’ or ‘faking’ might be a bit of a stretch. The YouTube video description of the demo mentioned earlier states the following: “For the purposes of this demo, latency has been reduced and Gemini outputs have been shortened for brevity.“ As for the MMLU claim, Google DeepMind’s website states that different prompting techniques were used.

While both sides have valid arguments, a tweet from the CEO of Perplexity AI Aravind Sriniva takes a balanced view: “Reality: Gemini is cool. The first model that genuinely is comparable to GPT 4. Real accomplishment. Especially that it was just a dense model. Marketing was overboard, but Deepmind is known for aggressive PR. Demos like the multimodal video in reality will be possible in less than a year.”

7 comments