From glue-on-pizza recipes to recommending "blinker fluid," Google's AI sourcing needs work.
See full article...
See full article...
Google’s “AI Overview” can give false, misleading, and dangerous answers
- Thread starter JournalBot
- Start date
Upvote
360
(364
/
-4)
I am not surprised considering LLM don't actually know anything. As for training with reddit and other social media...garbage in garbage out still holds true it seems.
Last edited:
Upvote
282
(285
/
-3)
- Add bookmark
- Featured
- #10
It should be noted that the issue is actually rarely the AI itself, but Google's ranking system and featured snippets.
In most cases where I've tested these "bad AI results", the actual problem is that the AI is just re-phrasing the top result that creates the featured snippets. As an example, the "How many rocks should I eat per day" one that's been making the rounds is actually just some gray-hat SEO of a featured snippet for a fracking company that cites an Onion article (SEO is complicated, folks).
So, the problem already exited in the form of featured snippets: Now it's just rephrased with AI.
Of course, Google has some real issues with handling these shifty SEO strategies, and it plays whack-a-mole constantly. I actually shut down one of my own websites because of Google not being able or willing to handle low-effort content farms that just gobble up content like mine and rewrite it using AI while doing a handful of additional techniques to edge out ranking over my original content.
In most cases where I've tested these "bad AI results", the actual problem is that the AI is just re-phrasing the top result that creates the featured snippets. As an example, the "How many rocks should I eat per day" one that's been making the rounds is actually just some gray-hat SEO of a featured snippet for a fracking company that cites an Onion article (SEO is complicated, folks).
So, the problem already exited in the form of featured snippets: Now it's just rephrased with AI.
Of course, Google has some real issues with handling these shifty SEO strategies, and it plays whack-a-mole constantly. I actually shut down one of my own websites because of Google not being able or willing to handle low-effort content farms that just gobble up content like mine and rewrite it using AI while doing a handful of additional techniques to edge out ranking over my original content.
Upvote
85
(116
/
-31)
- Add bookmark
- Featured
- #11
This is a great example of:
- garbage in, garbage out. Even the LLM says it's from a Reddit post.
- people having unrealistic expectations about LLMs. Perhaps this will convince everyone that they're parroting what they're fed and have no understanding or self consciousness.
- google shooting themselves in the foot. It's one thing to give a result like the Reddit suggesion as a link to the original post on Reddit. It's another one entirely to get it in this overview where it sounds like it's endorsed by Google.
Upvote
231
(235
/
-4)
Upvote
145
(146
/
-1)
What's that quote again about trust taking years to gain and an instant to lose?
Upvote
276
(276
/
0)
As more and more of the training data is fake posts and web pages generated by AI, this is going to get a lot funnier until it's not, and adults in charge wake up.
Don't forget Google is run on the social media model where engagement and delivering eyeballs is valued ahead of quality, facts, and editorial responsibility, so no wonder their first public facing AI is fraught with poor quality, fiction and almost no editorial oversight.
Don't forget Google is run on the social media model where engagement and delivering eyeballs is valued ahead of quality, facts, and editorial responsibility, so no wonder their first public facing AI is fraught with poor quality, fiction and almost no editorial oversight.
Last edited:
Upvote
185
(186
/
-1)
I listened to a podcast with Steve Gibson (GRC) this week and he shared an email he sent out 25 years ago when he first discovered google and hardly anyone knew what this new search was. He shared how incredible it was at that time and how it degraded over time. If this isn't the final straw what is?
Last edited:
Upvote
88
(91
/
-3)
And, yet, in Australia we have a major telecommunications company (and employer - it was the former gov-owned monopoly provider and still enjoys major market share) sacking 10% of its workforce "in an AI-driven "reset" of its enterprise arm" - https://ia.acs.org.au/article/2024/telstra-cuts-2-800-jobs-as-ai-takes-over.html
What could possibly go wrong?
What could possibly go wrong?
Upvote
115
(117
/
-2)
I am of the opinion (especially in this situation as a former firefighter myself) that fire code or any safety-critical use-case is inappropriate for general AI searches/summaries. Possibly Retrieval Augmented Generation (RAG) built around your use-case domain would be accurate enough, but that’s not what you are likely using. Your result validation is good, but for every person who does that, there are 10 who don’t, or who aren’t as diligent as you are. So please be careful about how you present this idea to others in your industry so people don’t get killed because of misuse/misunderstanding of this technology.
Last edited:
Upvote
180
(180
/
0)
And I'm not convinced these so-called adults will wake up or that they even exist. There is just too much money to be made, consequences be damned.
Upvote
117
(118
/
-1)
The problem, really, is money. Google makes a lot of it, but there's the possibility that they might make less money than they did before, or that someone else might make money that Google would like to make. That "not making money you feel you should be making" results in some serious maladaptive behaviour.
We talk about LLMs being just fancy pattern-imitation systems, but behaviourism drives a lot of what people do, too, and we're seeing a form of it here: executives get rewarded when stock price goes up, executives get punished when stock price go down, so executives, like an LLM AI, will follow the pattern of doing anything and everything that makes stock price go up, regardless of whether or not it'll make stock price down six months from now, will kill your brand, or will cause the rise of fascism in the west.
It's like the grey-goo issue, but for MBAs instead of nanomachines.
We talk about LLMs being just fancy pattern-imitation systems, but behaviourism drives a lot of what people do, too, and we're seeing a form of it here: executives get rewarded when stock price goes up, executives get punished when stock price go down, so executives, like an LLM AI, will follow the pattern of doing anything and everything that makes stock price go up, regardless of whether or not it'll make stock price down six months from now, will kill your brand, or will cause the rise of fascism in the west.
It's like the grey-goo issue, but for MBAs instead of nanomachines.
Upvote
195
(196
/
-1)
I for one am shocked that yet another company's glorified markov chain chatbot has turned out to be at best entirely fucking useless, and at worst, actively malicious.
Upvote
168
(174
/
-6)
This all just highlights that no part of any of this has ever been an “AI” and continuing to present it this way is misleading at best.
It’s just parroting back whatever it has been fed, including nonsense joke posts, or making up its own nonsense, and Google (and everyone else) is too focused on trying to get advertising dollars so they won’t even admit how inaccurate it can be, and thus how dangerous it is to present it as authoritative.
When it can independently analyze a source and determine if the information in it is correct, recognize a joke, and understand the context in which information is being given (and requested, it seems like it has a problem with that for some queries), then maybe they can consider calling it intelligent.
when your “AI” is a statistical model of the frequency and probability of certain words following other certain words in response to some words, and it has no way of actually understanding what those mean, you’ve got yourself a chat help bot. A big one, sure, but it has just been trained on a bigger dataset than the usual “one company’s useless outdated knowledge base articles”.
It’s just parroting back whatever it has been fed, including nonsense joke posts, or making up its own nonsense, and Google (and everyone else) is too focused on trying to get advertising dollars so they won’t even admit how inaccurate it can be, and thus how dangerous it is to present it as authoritative.
When it can independently analyze a source and determine if the information in it is correct, recognize a joke, and understand the context in which information is being given (and requested, it seems like it has a problem with that for some queries), then maybe they can consider calling it intelligent.
when your “AI” is a statistical model of the frequency and probability of certain words following other certain words in response to some words, and it has no way of actually understanding what those mean, you’ve got yourself a chat help bot. A big one, sure, but it has just been trained on a bigger dataset than the usual “one company’s useless outdated knowledge base articles”.
Upvote
134
(137
/
-3)
Blinker fluid? Is that available from the same place I get my elbow grease?
View: https://youtu.be/r9eS8PtfmRg
View: https://youtu.be/r9eS8PtfmRg
Upvote
49
(51
/
-2)
Are the people having unrealistic expectations of LLMs to which you refer actually CEOs of large tech companies?
Upvote
116
(116
/
0)
The problem self-magnifies too. As AI discourages and demonetizes human content creators, the corpus of training data becomes more and more AI-generated as you said. The “internet grey goo” concept isn’t new, but I’d take it even a step further and compare it to genetic diversity. We all know what happens to a population that lacks an injection of new genetics…what happens when people stop making new original content and all that’s left is just one big pile of stagnant content statistically rearranged over and over?
Last edited:
Upvote
165
(165
/
0)
Sundar Pichai just keeps knocking it out of the park. Google leadership grossly misunderstands the value proposition of Google Search. That's the problem with lack of visionary and insightful leadership. You keep chasing whatever new shiny thing others are doing. While not an Apple fan, I keep getting astounded at just how good they are at execution, focus and long term thinking. In comparison, Google just seems like a drunk, headless chicken staggering along.
Upvote
174
(176
/
-2)
To the question "where can I buy blinker fluid", Gemini, ChatGPT and Mistral said it was a joke. However, Phind started with "Blinker fluid, also known as brake fluid, is essential for the proper functioning of your vehicle's braking system."
Upvote
34
(34
/
0)
It's appreciated that so many of the links were to Bluesky, rather than TwiX as per usual.
Upvote
99
(104
/
-5)
Correct me if I'm wrong here, but isn't most LLM type AI just trying to make a correct looking answer based on the prompt given to it? If you feed it crap like Reddit and X posts then of course sarcasm and parody would be presented as fact.
Upvote
69
(69
/
0)
This is part of why AI is dangerous in my opinion. People currently say that it's fine so long as you verify the answer, but very few people will continue to do that long term if they don't spot immediate problems themselves.
Any safety measure that relies on a constantly attentive and educated human is not going to last - and always taking the effort to verify a result that is usually fine falls into that category.
I know someone who started using LLMs to get answers for their work contracts which they "always verify themselves first". But how many times does it have to be correct before you start wondering if verifying is really necessary?
This person says that they still always double-check the answer, but it actually took less than a week before they started to just kind of waive it past without checking at all. It's just how most people are.
In my experience, if there are no immmediate error with something, then as soon as the novelty wears off and people can't be bothered anymore, they just start trusting it. And most people are not very diligent in the first place.
If AI is only safe because the humans who use it are diligent and educated every time we use it then we're screwed.
Upvote
190
(191
/
-1)
Former firefighter, as well
I use it solely as a starting point. I.e. "What NFPA code references lumens per square foot for strobes in an office setting" to get me to the right NFPA code and section - I've heard enough horror stories about AI generated content that precisely zero of it is used in anything given to the client. I am of the opinion that, right now: AI is handy for speeding up the start-up portion of the process, but at least in my line of work: Then end result must still be 100% human generated - and verified.
Upvote
3
(24
/
-21)
Given that they're using reddit as a source, I'm surprised that we didn't get a bunch of dick jokes in with the blinker fluid suggestions.
We are nearly there.
My anecdote is about an ad on the Solitaire game that I play. An ad for "Fake Video Call" app shows up and any touch of the screen starts the install of this app. It's happened to me twice so far and is extremely irritating. I immediately uninstall it before it launches hoping it hasn't planted any malicious code on my phone. So I searched Google for "how to stop Fake Video Call app from being installed automatically". All of the results were about sources for fake video call apps. Who the hell wants to install a fake video call app? What exactly IS a fake video call app?
Upvote
88
(88
/
0)
As I've basically just mentioned in another comment: I agree, but I don't think the usage you describe is really possible for most people, and that's the problem.Former firefighter, as well
I am of the opinion that, right now: AI is handy for speeding up the start-up portion of the process, but at least in my line of work: Then end result must still be 100% human generated - and verified.
If you hand the average human a complete-looking report about something that appears accurate, and usually is, but then say "don't foget to completely redo it yourself checking every fact thoroughly", they just aren't going to do that. Not with the attention to detail they would have used if they had to do it all from scratch. Most people are going to rely on what these things tell them (or at least reduce their scepticism of it over time) whether they should or not.
Upvote
96
(97
/
-1)
In other news: Water is wet!
Everyone must have seen this coming. Every single LLM has these issues, and when you query it enough they can and will pop up. I reckon every single search query google receives should be enough....
Everyone must have seen this coming. Every single LLM has these issues, and when you query it enough they can and will pop up. I reckon every single search query google receives should be enough....
Upvote
-6
(12
/
-18)
More proof that these "AI"s aren't actually intelligent at all but rather simply regurgitate content it has sifted from the web and hopes it makes sense. It's so obvious that this whole "AI revolution" is in actuality the tech industry needing a new cash cow to milk. None of this is useful.
Upvote
85
(91
/
-6)