Google Gemini Advanced Review: Don’t Cancel Your ChatGPT Plus Subscription [CNET]

Google premium chatbot lags behind ChatGPT.

Basic info:

• Price: $20 per month
• Availability: Web or mobile app
• Features: Voice recognition, memory retention, high token count, cross-check with Google Search, connection to open internet with up-to-date information
• Image generation: yes

Two years into the AI revolution and Google is still fumbling. When Google was caught flat-footed by the release of ChatGPT in late 2022, that reportedly caused a code red within the company, and it rushed to get a competing AI chatbot out the door. At the time, it was named Bard, and the launch didn’t go smoothly. It immediately made errors, leading to a massive drop in Google’s stock price. 

Since then, Google’s AI efforts have continued to stumble, with the company having to respond to multiple embarrassments. This includes generating images in Gemini, renamed from Bard, of people of color portrayed as Nazis, or its AI Overviews feature in Search telling people to eat rocks or put glue on pizza. 

But these foibles have largely been limited to Google’s free AI offerings. With Gemini Advanced, which runs on the Gemini 1.5 Pro model, Google says it’s more powerful and has better understanding. At Google I/O earlier this year, the company touted that it will have a context window of 2 million tokens. That means you could paste a 3,000-page document and Gemini can parse through all of it. 

But being able to handle a lot of data doesn’t mean Gemini can use that data correctly. Even with the improvements to Gemini Advanced, we still found glaring errors in the paid version of Google’s chatbot. It’s a problem that plagued the free version as well. At least Gemini Advanced has the Double Check feature, which cross-references answers with Google Search results. It does raise the question: Why can’t Gemini integrate Google cross-checking within its answer stack? 

At $20 per month, which is the same price as ChatGPT Plus, Google’s failing to deliver on a competitor that can dethrone OpenAI.

How CNET tests AI chatbots

CNET takes a practical approach to reviewing AI chatbots. By prompting AI chatbots with real-world scenarios, like finding and modifying recipes, researching travel, and writing emails, my fellow reviewers and I aim to simulate what the average person might use them for. The goal isn’t to break AI chatbots with bizarre riddles or logic problems. Instead, we look to see if real questions prompt useful and accurate answers. However, as more advanced models are released, it’ll require more advanced-level questioning to see how AI chatbots can reason with complex and multilayered topics. See our page on how we test AI for more.

When using Gemini Advanced, keep in mind that the service automatically collects the information you put into its system, so be mindful of giving the service any personal information. Google recommends not feeding Gemini any sensitive information whatsoever. For more information, see Google’s privacy policy.

Shopping

It’s reasonable to expect that when using a paid AI chatbot for shopping recommendations it should get basic information correct. Well, that’s expecting too much from Google Gemini. Or maybe that’s expecting too much from AI chatbots in general, at least for recently released products.

When trying to compare the newly released Motorola Razr Plus 2024 and the Samsung Galaxy Z Flip 6, Gemini got key bits of information wrong, bits that could greatly affect your purchasing decision. This includes basic information, like claiming the Razr Plus has a smaller display than the Z Flip 6, when in reality it’s the other way around. Or that the Z Flip 6 is bulkier than the Razr when folded, whereas it’s actually slimmer. Gemini even failed to get the processor specifications correct on both devices. 

Thankfully, Gemini does have a double-check option that lets you cross-reference Google Search to ensure the answer is correct. The problem is, how would a person know to double check? Is it best to double check on all queries? If the user doesn’t know the information is wrong, then how will they know to click on Double Check?

Granted, when asking ChatGPT similar questions about the Razr Plus and Z Flip 6, it too got lots of information wrong. Even though ChatGPT 4o and Gemini Advanced are connected to the open internet, both systems need some fine tuning for how they parse through relatively recent bits of information. 

As for comparing products that are a year old, Gemini did a much better job of getting information right. I asked it to compare the LG OLED C3 and G3 TVs, and Gemini got all the little factoids correct. This suggests that it takes time for AI chatbots to ascertain and funnel information into their datasets. So, whether you’re using Gemini Advanced or ChatGPT Plus, for newer items, a Google Search might be your best bet.

Recipes

All chatbots, including the free ones, do a good job of conjuring up recipes. Some might perform better than others but all are adequate. 

When reviewing free AI chatbots like ChatGPT 3.5, Claude and even the base Gemini, I had no issues with having recipes generated. Weirdly, Gemini Advanced glitched out for what should have been a straightforward recipe query. Why it did so, I’m not sure.

When I tried asking it again, it began generating the recipe only to then delete it and say, “I’m not programmed to assist with that.” Bizarre.

I thought maybe Gemini has an aversion to South Asian food. I then asked it to make me a Thai-inspired tres leches cake, mixing the flavors of Mexico and Southeast Asia. Even here, it began generating a page-long recipe for it to suddenly disappear and say it’s unable to process that. After a second try, it finally generated a recipe. 

The recipe itself seemed fine but a bit lacking. The only Thai ingredients were Thai tea and some shredded coconut. Some lime zest or cardamom would have been nice additions.

So when Gemini Advanced works, it’s a decent recipe builder. But it might be easier and better to defer to a dedicated recipe website.

Summarizing articles

Whether it be Microsoft, OpenAI or Google, all have touted AI’s capabilities to synthesize and summarize information. Long emails can quickly be condensed to a few sentences with the key parts pulled out. A research paper can be shrunk down to one page for quick viewing. Sure, a summary won’t give you the full scope of what a 30-page research paper would. But in a pinch, it’s handy. 

At the very least, AI chatbots should do a decent job of summarizing a short article to get all the key points. Weirdly, this has been a difficult task for the free AI chatbots. They’d either run into character limits or get some of the underlying points but miss the bigger picture. 

Just like with my recipe test above, Gemini failed to summarize an article I wrote, saying it can’t access CNET.

But when I simply pasted the link without prompting Gemini to “summarize this article for me,” it actually did return a few bullet points. Why the shift in response between prompted and promptless is lost on me. 

While I technically didn’t prompt Gemini to summarize my article in this instance, it still did so anyway. Honestly, the breakdown isn’t bad. It got the main points and even got the crux of the piece. 

According to Jon Henshaw, CNET’s vice president of audience, our site isn’t blocking Google from crawling it. CNET doesn’t allow Google to train its LLMs with our journalism, however. 

Travel

As a travel planning tool, Gemini Advanced does a great job of building a robust itinerary. The test isn’t notably difficult: creating a three-day travel itinerary to Columbus, Ohio. While it’s not the most traveled-to destination, it’s still unconventional enough where we feel it would mildly challenge AI chatbots.

The advantage with Gemini Advanced is that it’s connected to the open internet. When running this test on free AI chatbots, like ChatGPT 3.5 to Claude, which have older datasets, sometimes it’d pull up restaurant recommendations for locations that have since closed. This hasn’t been a problem for Gemini Advanced in our testing.

What’s also handy is that Gemini Advanced is linked with Google Maps. Meaning, I could ask Gemini for retro video game store recommendations and it’d pull up a map of five stores around the Columbus area. It gives Gemini Advanced a leg up over the competition.

The inclusion of articles to sites like Tripadvisor, along with pictures so you could see what things look like without leaving Gemini, was certainly an appreciated feature.

Now, since this isn’t the most demanding test of an AI chatbot, the free version of Gemini produces similar results and also links to Google Maps. While Gemini’s travel recommendation functionality is solid, it’s not worth the $20 per month for this function alone. 

High-level thinking

When reviewing ChatGPT 4 earlier this year, we tried to push its cognitive capabilities by asking philosophically obscure questions requiring it to juggle different schools of thought from past philosophers while simultaneously putting it in a modern technological framework. You could say it was a bit absurd. 

Given the obtuse wording, we were surprised at how well ChatGPT 4 was able to handle our jumble of philosophical jargon. 

Gemini Advanced, in comparison, reads like a Bart Simpson book report, with its only knowledge of the source material being the cover art alone. What’s worse, Gemini instead suggested what I should Google search instead to learn more. The constant return to Google Search sums up the experience with Gemini Advanced rather succinctly.

If you must spend $20 on an AI chatbot, Gemini Advanced isn’t it

Like the free version of Google Gemini, the paid Gemini Advanced isn’t worth your time or attention. It’s bizarre how poor Gemini Advanced is when Google is the company that funded much of the neural network technology powering the AI revolution. 

While Gemini Advanced works adequately in answering basic questions, it starts to fall apart when pushing it for novel information analysis. Heck, it’ll start falling apart when asking basic questions too. The fact that Gemini glitched out multiple times when generating something as simple as a recipe makes the product feel unfinished. It echoes Google’s past blunders with AI and feels as if the company pushed out a product before it was ready. 

To put it bluntly, don’t waste your money on Gemini Advanced. ChatGPT Plus is simply better. It’s smarter and more accurate. The only area where ChatGPT really loses out against Gemini Advanced is its token count. You can paste in text only up to 4,096 tokens, or about 3,000 words. That pales in comparison with the 1 million context tokens that can be entered into Gemini Advanced. 

That said, most people won’t be pasting entire novels into AI chatbots. Unless you have one particular use case that only Gemini can fulfill, ChatGPT Plus is the better option on all counts.