Even though I’m relatively wary of AI answers, I still occasionally turn to AI tools to quickly understand certain topics. This is especially true when I use Brave Search to get an answer to a question.
But there’s a reason I take these answers with a pinch of salt. Often when I ask AI tools about something I already have a deep understanding of, I start to notice flaws in their answers — and sometimes, outright hallucinations or bad advice.
Related
I use ChatGPT, Claude, Perplexity, and Gemini daily — here’s the only one worth paying for
One stands above the rest.
AI answers sound good until you start picking up the mistakes
Sometimes the advice is just downright bad
AI chatbots have a way of being confidently incorrect — presenting dubious information as completely true. I’ve even had to argue with tools like Gemini when I know an answer is wrong. Eventually, the chatbot relents, but I could’ve walked away having taken the first response as the truth.
This happens across topics, from low-stakes image identification to queries about subjects I know well. Recently, I decided to replicate this by asking ChatGPT, Claude, and Gemini about whether they would recommend a phone with a low PWM dimming rate to someone with migraines.
For context, low PWM dimming rates are associated with screen flicker at lower brightness, which may affect certain sensitive individuals and cause headaches, migraines, nausea, and eye strain. All the AI tools correctly noted that while PWM dimming rates affect people differently, someone with chronic migraines should err on the side of caution and choose a phone with DC dimming or a high PWM dimming rate.
But when it came to product suggestions, the cracks started to show. ChatGPT overlooked key specs based on its own suggestions, only adjusting its ranking when I pointed out that a lower-ranked phone had a top-tier trait: DC dimming. It then corrected its ranking. But its shortlist also included the Samsung Galaxy S26 Ultra, which it actually recommended against buying later in its own response.
Quizzing Gemini similarly revealed a decline in reliable answers. Gemini suggested that someone with chronic migraines turn their phone up to full brightness and then use third-party software or built-in features to lower the brightness. While this solution could eliminate flicker to some extent, it overlooks a key consideration: bright screens worsen and trigger migraines. Even with a feature like Extra Dim or Reduce White Point enabled, 100% brightness is still too bright. I even tested this on my Samsung Galaxy S23 Ultra to confirm it.
Third-party screen-dimming apps can also pose a security risk, as they let the app apply an overlay on your screen. This is something the AI failed to mention.
For product recommendations, Gemini suggested smartphones that were a few generations old, despite the prompt including no price range or time frame. For example, it suggested the Honor 200 series, even though Honor’s latest number series is the Honor 600. In fact, it recommended only phones from 2023 to 2024 for no apparent reason.
Meanwhile, Claude just suggested brands rather than specific phone models. However, at least the information was accurate, even though it was vague.
The answer depends on the chatbot
For questions with objective answers, you’d expect AI tools to reach similar conclusions. But sometimes AI tools will steer you in completely different directions.
I noticed this recently when trying to propagate some new plants from my succulents. I asked whether I could use a flower stem to propagate new pups. Gemini confidently said I could do this and gave me instructions. A few days later, when I wanted to remember the steps, I asked Brave Search how to propagate succulents from the flower stem. Instead, it told me this was not possible.
This contradiction is not a one-off quirk either. Today, when I quizzed ChatGPT, Claude, Gemini, and Brave Search about this propagation method, the chatbots gave conflicting answers.
Claude and Brave Search took the stance that, generally, you can’t propagate succulents this way. But Gemini and ChatGPT said this was possible. Gemini didn’t seem to doubt it, but ChatGPT noted it’s only sometimes achievable.
Sources and models matter
Some tools perform better than others
Whenever I prod AI tools for answers, Claude and Brave Search tend to perform the best. Part of this is because the tools can express uncertainty, but also because of the general quality of sources.
That said, they don’t avoid faltering. Brave Search, especially, can prioritize sources that aren’t necessarily authoritative. For example, if I ask the search engine a really niche question, it often cites Reddit as its source. I’ve had responses that essentially rely on one Reddit comment as the citation.
This highlights how sources and model behavior can shape answers. Even NotebookLM, which tends to hallucinate much less than other AI tools, can go awry with the wrong sources.
To give the tools credit, however, they did explain a few topics well when there were many sources available. But it’s when you get into the nitty-gritty that they often seem to falter. Claude, though, remains the tool that impresses me the most.
Related
I borrowed Claude prompts from Anthropic engineers and immediately stopped wasting time on bad ones
The best prompts come from the team.
It’s a reminder to fact-check the answers AI gives you
There are constant disclaimers that AI can get things wrong, yet we still see people use AI answers as a single source of truth. But when you really start testing their knowledge, it becomes clear how often these answers overlook key factors, contradict themselves, or just provide low-quality information.
Using AI answers for low-stakes queries won’t backfire too much. You might just waste some time trying to grow a succulent the wrong way. However, if you want reliable information on topics you’re not sure about, check sources and get answers from authoritative outlets.
For more insights, you can check the full article Here.
“`
