News reporters and editors have their biases. Book authors have their biases. Scientists and research papers have their biases. Search engines have their biases. Google too.
All human-created systems have biases shaped by the environments, social norms, education, traditions, etc. of their creators and managers.
So, the concepts of "objective truth" and "reputable" need to be analyzed more critically.
They seem to be labels given to sources we have learned to trust by habit. Some people trust newspapers over TV. Some people trust some newspapers over other newspapers. All of it often on emotional grounds of agreeability with our own biases. Then we seem to post-rationalize this emotion of agreeability using terms like "objective truth" and "reputable".
Is Google search engine that leads to NY Times or Fox News or Wikipedia and makes us manually choose sources as per our biases "better" than Google's Gemini engine that summarizes content from all the above sources and gives an average answer? (Note: "average answer" as of current versions; in future, its training too may be explicitly biased, like Grok and DeepSeek did).
Perhaps we can start using terms like "human sources of information" versus "AI sources of information" and get rid of the contentious terms.
Then critically analyze whether one set of sources is better than the other, or they complement each other.
I'm going to hold them to the same standard no matter if they use crappy sources, plagiarize, or hallucinate on their own. If someone asked, when and if I am in a position where I have to tell them, I would remind them that LLMs prioritize their own confidence over correctness.
LLMs aren't a special case to me. Glue doesn't belong on pizza and you shouldn't eat one rock a day but we've been giving and getting bad advice forever. The person needs to take ownership for the output and getting it right, no matter the source, is their responsibility.
Ask them to tell the LLM it's wrong... then when it goes "You are absolutely right!" to challenge it and say that it was a test. Then when it replies, ask it if it's 100% sure. They'll lose faith pretty quick.
This is an oft-repeated meme, but I’m convinced the people saying it are either blindly repeating it, using bad models/system prompts, or some other issue. Claude Opus will absolutely push back if you disagree. I routinely push back on Claude only to discover on further evaluation that the model was correct.
As a test I just did exactly what you said in a Claude Opus 4.6 session about another HN thread. Claude considered* the contradiction, evaluated additional sources, and responded backing up its original claim with more evidence.
I will add that I use a system prompt that explicitly discourages sycophancy, but this is a single sentence expression of preference and not an indication of fundamental model weakness.
* I’ll leave the anthropomorphism discussions to Searle; empirically this is the observed output.
If you have 10,000 people flipping coins over and over, one person will be experiencing a streak of heads, another a streak of tails.
Which is to say, of a million people who just started playing with LLMs, a bunch of people will get hit or miss, while one guy is winning the neural net lottery and has the experience of the AI nailing every request, some poor bloke is trying to see what all the hype is about and cannot get one response that isn’t fully hallucinated garbage
Sure, but that doesn’t explain the volume of these complaints. I think the more likely answer is the pitiful sycophancy of some models as demonstrated in BSBench.
Yes, somehow. I have been dealing with an awful lot of people who basically have what are theoretically logic degrees who suddenly just take LLMs at face value, or quote them to me like that actually means anything. People I formerly thought were sane.
I don't mean to put words in your mouth but from what I've seen, in person but mostly online, but the "problem" (and I put that in quotes because I don't even know what to call it... it seems deeper than a mere "problem") is that they quote them as if they are autonomous, sentient beings.
People just don't like to be played for fools. Perhaps us giving into this is progress? I'd give a big ol' "fuck you" to anyone who claims it is, but I'm also pretty old.
I am not sure I would even say "believe", I would think of it more as short-circuiting our critical thinking. I think it taps into something at the core of our tribal instincts. It was famously present in even basic systems like Eliza. And it's not just machines... The same tricks are used by conmen, politicians, and psychopaths, which is more negative than I intend. Even with good intentions and positive outcomes, I feel we need to remember that we drive it, not the other way around.
I’ve seen some people quote AI like you’re saying. However, when I preface something with “ChatGPT said…”, my intention is to convey to the listener that they should take it with a grain of salt, as it might be completely bull shit. I suppose I should consider who I’m talking to when I make that assumption.
it’s a slightly orthogonal problem to using the active voice of “XYZ says…”, it’s treating the text continuation engine as an “other” that may know better than they do, playing into sci fi conceptions of AI having its personal positronic brain or whatever, having its own ideas and deciding to carve a horse out of driftwood.
It’s not quite anthropomorphizing either that’s the issue, need a word for “treating it as tho it were a machine conscious that exists alongside humanity*”, how does cyborgropomorphizing sound
* and not merely a markov chain running in Sam Altman’s closet
Surely the correct conclusion is to question the value/veracity of those degree issuing institutions and rituals?
And if you previously were unaware of the insanity and irrationality passing under the surface of such human activity, I guess it can come as a bit of a shock :)
It happened with science, politics, traditional media, history books, "good engineering practices" applied to IT, OOP,tdd,DDD,server side rendering, containerization...
Literally every bullshit shilled to the moon is accepted without second guessing and you would be without a job, in an asylum, for questioning 2 of them in a row.
Why is it different now? EVERYTHING is bullshit, only attention matters. And craftsmanship.
I don't think this has anything to do with sanity. This has to do with people for seeking self confirmation instead ot disproval.
For pretty much everything there is a conspiracy theory out there claiming the opposite, and these types usually started out searching the internet for someone else who believes the same that they did at the time.
But, as we all know, this technique will eventually lead to overfitting. And that's what those types of people have done to themselves.
Well, and as lack of education is the weakness of democracy, there's a lot of interested parties out there that invest money in these types of conspiracy websites. Even more so after LLMs.
Whoever controls the news controls the perpetual presence, where everything is independent of the forgotten history.
Yes, I think AI bots are more compelling to some people. They break the concept of judging information by its source because they obscure the source. But at the same time they are trained on a lot of reputable sources and can say a lot of very smart things, just at other times they say complete BS. But they are really good at making things sound plausible, that's essentially how they work after all.
I would argue for many people social media and news aggregators do the exact same thing. People site Instagram or TikTok as the source of their data in the same way others site ChatGPT
Not that I've had to deal with this specifically, but I have noticed how the input phrasing in my prompts pushes the LLM in different directions. I've just tried a quick test with `duck.ai` on gpt 4o-mini with:
A: Why is drinking coffee every day so good for you?
B: Why is drinking coffee every day so bad for you?
Question A responds that it has "several health benefits", antioxidants, liver health, reduced risk of diabetes and Parkinson's.
Question B responds that it may lead to sleep disruption, digestive issues, risk of osteoporosis.
Same question. One word difference. Two different directions.
This makes me take everything with a pinch of salt when I ask "Would Library A be a good fit for Problem X" - which is obviously a bit leading; I don't even trust what I hope are more neutral inputs like "How does Library A apply to Problem Space X", for example.
Again a model issue. At the risk of coming off as a thread-wide apologist, here are my results on Opus:
Good:
> The research is generally positive but it’s not unconditionally “good for you” — the framing matters.
> What the evidence supports for moderate consumption (3-5 cups/day): lower risk of type 2 diabetes, Parkinson’s, certain liver diseases (including liver cancer), and all-cause mortality……
Bad:
> The premise is off. Moderate daily coffee consumption (3-5 cups) isn’t considered bad for you by current medical consensus. It’s actually associated with reduced risk of type 2 diabetes, Parkinson’s, and some liver diseases in large epidemiological studies.
> Where it can cause problems:
Heavy consumption (6+ cups) can lead to anxiety, insomnia……
Clickbait journalists answers like that, experts mostly don't. But it does make sense it mimics clickbait journalists more since it was trained on the internet.
That's true and fair, and re-reading OP it doesn't address hallucinations exactly either. I was more thinking of it as a toy example for non-tech folk (grandma?) to see that what and how you ask LLMs matters in how the sycophancy will come out in the response. There may be better ways to demo that though :shrug:
Are you talking about people who will still insist the LLM was correct even after being presented with evidence to the contrary, or people who don't EVER bother double checking answers they get out of said software since they assume it to be true?
The same way that I handle anyone who blindly trusts anything on the internet. Could be an LLM, TikTok or YouTube video, Wikipedia article, news article, whatever.
It usually involves some form of "well, no, hold on..."
A search isn’t going to lead them to a “reputable source”, it’s going to lead them to ad filled SEO garbage, because it’s not 2004 anymore and thousands of Google employees have been working for two decades to ruin the Internet.
I’ll take LLMs any day over what search and the rest of the Internet has turned into.
Right, I think (hope) the OP meant not to emphasize the "search" in the sentence, but the "reputable source". Of course a Google search now is much worse than an AI search.
And it is the ultimately the reputable source that matters, and whether the person actually read it and checked that the details matched the summary (be it human abstract, LLM-generated, or otherwise).
I treat people who blindly believe an LLM the same way I treat people who blindly believe a religion or a political ideology or medical advice from Instagram.
If they ask what I think, I tell them.
If they don't want my opinion I keep it to myself.
> They have a question that would be very well answered with a search leading to a reputable source
Can you give an example of what kind of question you mean here?
Given that most people's idea of a reputable source is whatever comes up on the first page of Google or YouTube, I think we should use that as the comparison rather than dismissing LLM results. And we should do some empirical testing before making assumptions, otherwise we're just as bad as the people we are complaining about.
Whatever results we get, the real problem is that most people's ability to verify information was not good before LLMs, and it's still not good now.
So now you're dealing with LLM hallucinations, and before you were dealing with the ravings of whatever blogger or YouTuber managed to rank for this particular query.
Sure LLMs make mistakes, but have you looked at the accuracy of the average top search results recently? The SERPs are packed with SEO-infested articles that are all written by LLMs anyway (and almost universally worse ones than you could use yourself). In many cases the stakes are low enough (and the cost of manually sifting through the junk high enough) that it’s worth going with the empirically higher quality answer than the SEO spam.
This of course doesn’t apply to high-stakes settings. In these cases I find LLMs are still a great information retrieval approach, but it’s a starting point to manual vetting.
Unless they are someone that values your opinion there's nothing you can do other than move on.
Some comments here equating it to people who blindly believe things on the internet, but it's worse than that. Many previously rational people essentially getting hypnotized by LLM use and loosing touch of their rational thinking.
Simple. I became one of them. Ultimately, using an AI is a new skill, but you have to treat it like another person that sometimes bullshits you. That's why you leverage agents to refine, do research, and polish.
Ask AI to cite sources and then investigate the sources, or have another agent fact check the relevancy of the sources.
You can use this thing called ralph that let's you burn a lot of tokens at scale by simply having a detailed prompt work on a task and refining something from different lenses. It too AI about an hour to write: https://nexivibe.com/avoid.civil.war.web/
I do this on things that I know very well, and the moment I let it cook and iterate, collect feedback, the results become chef's kiss.
The agentic era that we are in is... very interesting.
It's incredible watching people determine that outsourcing their thinking and work to what has been generously described as a junior coworker is a new 'skill'. Words are losing their meaning, on multiple levels.
counterpoint: if I have to treat the computer like a person, what's the point of talking to a computer in the first place? Particularly when there are so many other systems that can provide answers without the runaround
Now we watch this viewpoint proliferate thousands and thousands of times over, even if it's less commonly stated so baldly, and yet people still wonder where the doomer viewpoints stem from?
I don't fight them on it. I just ask "where did that come from?" and suggest checking a real source. Most people aren't trying to be wrong, they just want quick answers. If you show them how to double check without making it a big deal, they usually get it.
I think LLMs are fine for a "first pass" on a topic, but if I am researching something, I want a primary source rather than just the LLM-generated output. Do they have the primary source?
tell them what to prompt the AI with to get the correct results. I've seen a number youtube shorts lately doing this, where some scientist gets "refuted" by some random person based on an LLM result, they then sit with the LLM and ask the same question, get the same wrong answer, then follow it up with a clarifying question, which then the LLM realizes its mistake and gives a better answer.
And then ask another question, and the LLM changes its mind again ("are you sure?").
It's not actually realizing anything so much as it's following your lead. Yes, followup questions can help dislodge more information, but fundamentally you can accidentally or on purpose bully an LLM to contradict itself quite easily, and it is only incidentally about correctness.
Explain that the models are compressed with a lossy compression, and point out that every so often, an answer will be pulled from a section of the model space which has compression errors.
I hear you,
But my understanding is LLM can consume multiple sources of information and deduce what is the truth better and more accurately than a human clicking on multiple Google links and veryfying information and sources.
Simply prove them wrong (earnestly and in good faith). When they realise the LLM is fallible, they'll learn to be skeptical of it without you needing to teach them that specific lesson.
At work I had this kind of discussion on a conference call, someone looked something up about a internal company policy and it came back with a hallucinated wrong result.
So I said, don't ever trust the output of an LLM without verification. However this caused me some hassle with the AI adoption manager. We have minimum-use AI KPI's for employees and he asked me to stop saying these things or people will use it less.
In the end I just hated the company a little bit more. I'm just sick of fighting against idiot. And he does have a point, our leadership is pretty crazy about the AI hype, they want everyone to be on it all the time. They don't seem to care whether it adds value or if it even detracts.
I have a feeling this is like telling people "don't touch a live wire" and the more direct experiental "I won't touch a live wire again" lesson: People need to experience being hallucinated at, within their comprehension, and at best can be told about Gell-Mann Amnesia.
I doubt you can stop them from asking machines for answers. What you can do is aide them to learn how to distrust the answers competently, but outside their field of knowledge, applying skepticism is hard.
The irony of Gell-Mann Amnesia is that Michael Chrichton, who is said to have named it, suffered from it badly: Wrote well within his field, misapplied sciences to write well outside it, and said things which were indefensible.
Honestly, the kind of people doing that is probably better served by AI (currently).
I'm saying that because they were not going to be critical of the search results, and google is not exactly showing objective truth in the first positions nowadays.
Thanks for this. I was in the camp of trust the LLM but y’all have made valid points. After discussing with ChatGPT, it agree there are some areas where it should not be trusted as accurate, but it said with historical facts like the holocaust it should be high. Idk, perhaps we need labs to produce a chart of it level of trust deserve to certain topics
LLMs are trained to at least try not to be dangerous and stupid, which is more than you can say for the executive branch and half the country. On balance, I'd trust a SOTA LLM model blindly before I'd trust the average person blindly.
Its everywhere now its becoming a real problem in every corner of the internet and in the real world. People are using hallucinated legal cases in lawsuits, they are generating images to create fake events, they are using AI to write their CVS and just about everything you can imagine. People are having to wade through all this slop professionally, calling it out and pointing out the mistakes doesn't seem to help, the people using this stuff believe the AI is correct no matter what you say or do.
Like most things that go mainstream it will take a good while before people understand, by which point they will have learnt a lot of things that aren't true and they will never let them go. We might get a healthy use of current AI at some point in the future or if the product drastically improves.
All you can do now is hold them to the same standard you normally would, if you catch them lying whether an AI did it or not its their responsibility and you treat them accordingly.
Accept this this is going to continue to happen, ask yourself if it’s something in your control or not, and try to find a way to enjoy the ride. It’s going to be bumpy, as we’re going through trust issues outside of just LLMs as a society right now.
However, if I notice a friend is about to harm themselves in some way I’ll pull open their ChatGPT and show them directly how sycophantic it is by going completely 180 on what they prompted. It’s enough to make them second guess. I also correct people who say “he or she” when referring to an LLM to say “it” in dialog, and explain that it’s a tool, like a calculator. So gentle reframing has helped.
Sometimes I’ll ask them to pause and ask their gut first, but people are already disconnected from their own truths.
A friend of mine severely injured her leg, especially knee, and went through a surgery. She said that she had a rehabilitation plan for the next 6 months. Guess what, from Gemini. I just told her just listen to her doctor.
I didn't tell her why LLMs can make mistakes or hallucinate because I thought that she would not appreciate my mansplaining.
Looking forward though, my boring answer would still be education. It is going to take time. But without understanding LLMs, they will not be easily persuaded.
LLMs give you sources now even if you don’t ask for it. If you don’t like it you can ask for more reputable sources. What kind of 2023 question is this?
"Tell me about all the potential pitfalls of blindly trusting LLM output, and relate a couple or three true stories about when LLM misinformation has gone badly wrong for people."
At the very least, I'm glad most people finally recognize LLMs are being used as a political weapon against education. It's the same old power struggles as ever.
These people may be idiots who are impossible to reason with, but at least for now the LLMs have not been completely driven into the ground by SEO. They might actually be getting a taste of what it feels like to not be an idiot. I'm happy for them, but they'll snap out of it when their trust is broken. It's probably sometime soon anyway.
The people who trust bad information from LLMs are the same people who trusted bad information from search results and new articles, it just takes them less time to get bad information.
There is no point to argue with stupid people. It's the same people who support their "opinion" with internet articles (like that means anything), mainstream media (hard to find bigger deceivers), or social media posts (that's arguably the worst).
Now they got another "God" in LLM.
How to deal? Just ignore. There is way more stupid people with stupid opinions than we can possibly estimate.
They do not have a soul, they are NPCs incapable of reasoning. I don't mean lazy, incapable is literally what they are. Logic escapes them.
When they say llms are conscious, and fully intelligent, them are comparing to themselves. If you think about it, they are right to say AGI is here, if the bar is the average human being.
If you contemplate this fact for a moment, and start pondering it could be true, your life would change forever. Most beings just do not have a singular perspective, cannot reason, do not have a taste, cannot appreciate someone else's singular perspective. They also do not appreciate art for the same reason.
I am sorry, truly.
Just let them be. They would kill you before admitting they are forever stuck in Plato's cave.
The worst are the ones who say "what if human brains are also just like LLMs". If you've thought about this for any length of time, it's very obvious that we aren't.
>If you've thought about this for any length of time
If you full throttle a BWM1000 S RR for a split of a second,at 30mph first gear, it will self eject beneath you. If you do that, for any length of time, you're dead.
Do the same on a 50cc motorbike and not much will happen. Even for extended periods of time, not much would happen.
You could hold it down until you run out of fuel or the universe gets cold and die, not much would happen.
You see, it's not that they are lazy. Or they haven't put any amount of time into understanding how llms operate.
Again I am sorry, most people are not capable, at all, of understanding what is happening at inference time. Most developers, nerds, hackers, who do understand how computers operate, cannot really grasp the basics of what an llm is or what the f is going on. Imagine the average guy, your lawyer, the MBA type of person.
Look I get your sentiment. Sometimes it feels like you're the only thinking, conscious being. Surrounded by beings who fundamentally cannot understand that A –> B does not imply B –> A. Beings that say things that are so obviously non-sequiturs or contradictory.
But calling people NPCs is the most NPC thing you can do. There is more to people than logical reasoning and these things often impede or completely block reasoning. Very intelligent people sometimes say the most grotesque things. People turn mad and mad people sometimes get their head set straight.
Sometimes it's not so much about the pure ability to reason but the goal of that person and whether they see understanding something or trying to understand it as helpful towards that goal.
I do agree though that the more intelligent someone is the less likely it is that other things will block their intelligent ability and the harder it is for them to fool themselves into believing absurd nonsense and to blind themselves from apparent truth.
Sometimes after talking with someone – or rather trying to but ending up only talking to them because they just do not manage to understand what I'm saying or to engage with it in any way – I wonder how they manage to get through every day life as that requires solving way more complex practical problems. Yet they do.
I firmly believe that every single person on this entire planet has a depth to them that far, far exceeds anything an LLM could even begin to approximate. I'm sorry you're in a position that you can't see that at all - that each and every one of them feel happiness and sadness and love and hate and fear and rage and inspiration and passion and are utterly human. I hope you see it someday.
I agree with you, every living being is beautiful in their own way. And good on their own way.
But we not very well equipped to follow the rules of logic. The most capable of us can follow a very limited number of logical threads, with very limited steps.
I don't consider it dehumanizing because attaining true knowledge is not necessarily human or natural for the human apparatus to process. Given enough strain and effort, we can only dabble in the elevated grounds of information and logic. Most people cannot, that is okay.
I am not comparing llms to humans in a broad sense, but specific to reasoning, llms are complete shit and so it is the average human. I am not impressed by llms if that was your impression and I am not saying the average human is inferior or deserve suffering lol
There are two kinds of fools in the world: the kind who ask a search engine and believe the first reputable source they see, and the kind who ask an LLM and believe the first response that has a reputable citation.
Yesterday, I was praying to ChatGPT and asking for guidance on my car washing problem. Through its holy scripture, it suggested me to walk to the car wash to improve my fitness. When I arrived and found the absence of my car to be a true hindrance for washing, it occurred to me that I should have pondered the scripture more carefully to identify its true meaning.
I treat the LLM like a diety. Every sane person understands well enough that the Bible is not to be taken literally. And then when someone talks about using LLMs, I always rephrase that as prayer.
> a reputable source
News reporters and editors have their biases. Book authors have their biases. Scientists and research papers have their biases. Search engines have their biases. Google too.
All human-created systems have biases shaped by the environments, social norms, education, traditions, etc. of their creators and managers.
So, the concepts of "objective truth" and "reputable" need to be analyzed more critically.
They seem to be labels given to sources we have learned to trust by habit. Some people trust newspapers over TV. Some people trust some newspapers over other newspapers. All of it often on emotional grounds of agreeability with our own biases. Then we seem to post-rationalize this emotion of agreeability using terms like "objective truth" and "reputable".
Is Google search engine that leads to NY Times or Fox News or Wikipedia and makes us manually choose sources as per our biases "better" than Google's Gemini engine that summarizes content from all the above sources and gives an average answer? (Note: "average answer" as of current versions; in future, its training too may be explicitly biased, like Grok and DeepSeek did).
Perhaps we can start using terms like "human sources of information" versus "AI sources of information" and get rid of the contentious terms.
Then critically analyze whether one set of sources is better than the other, or they complement each other.
I'm going to hold them to the same standard no matter if they use crappy sources, plagiarize, or hallucinate on their own. If someone asked, when and if I am in a position where I have to tell them, I would remind them that LLMs prioritize their own confidence over correctness.
LLMs aren't a special case to me. Glue doesn't belong on pizza and you shouldn't eat one rock a day but we've been giving and getting bad advice forever. The person needs to take ownership for the output and getting it right, no matter the source, is their responsibility.
this. llm's aren't that special, access _maybe_, but there's plenty of access to terrible rumor mills.
i love you
Ask them to tell the LLM it's wrong... then when it goes "You are absolutely right!" to challenge it and say that it was a test. Then when it replies, ask it if it's 100% sure. They'll lose faith pretty quick.
This is an oft-repeated meme, but I’m convinced the people saying it are either blindly repeating it, using bad models/system prompts, or some other issue. Claude Opus will absolutely push back if you disagree. I routinely push back on Claude only to discover on further evaluation that the model was correct.
As a test I just did exactly what you said in a Claude Opus 4.6 session about another HN thread. Claude considered* the contradiction, evaluated additional sources, and responded backing up its original claim with more evidence.
I will add that I use a system prompt that explicitly discourages sycophancy, but this is a single sentence expression of preference and not an indication of fundamental model weakness.
* I’ll leave the anthropomorphism discussions to Searle; empirically this is the observed output.
Claude Opus 4.6 is the best possible model to use in this test, with the least sycophancy. OpenAI and Gemini models are bad in comparison.
ChatGPT thinking models are very good; the instant model is bad. Gemini is always desperate to find an answer, and will give you one no matter what.
If you have 10,000 people flipping coins over and over, one person will be experiencing a streak of heads, another a streak of tails.
Which is to say, of a million people who just started playing with LLMs, a bunch of people will get hit or miss, while one guy is winning the neural net lottery and has the experience of the AI nailing every request, some poor bloke is trying to see what all the hype is about and cannot get one response that isn’t fully hallucinated garbage
Sure, but that doesn’t explain the volume of these complaints. I think the more likely answer is the pitiful sycophancy of some models as demonstrated in BSBench.
Can you share your system prompt?
I'm seeing the described behavior with whatever the default system prompt is in Claude Code.
I tried to fool claude sonnet with confidence and it failed.
https://claude.ai/share/47145af0-47d1-451b-813c-131ec48e7215
Maybe it is possible with a more complex or subjective question.
Is this any different than people who believe random things they read on sketchy news sites or social media?
Yes, somehow. I have been dealing with an awful lot of people who basically have what are theoretically logic degrees who suddenly just take LLMs at face value, or quote them to me like that actually means anything. People I formerly thought were sane.
I don't mean to put words in your mouth but from what I've seen, in person but mostly online, but the "problem" (and I put that in quotes because I don't even know what to call it... it seems deeper than a mere "problem") is that they quote them as if they are autonomous, sentient beings.
The problem is that LLM output looks like a human conversation. People believe it.
Which is more believable?
“The sky is filled with a downpour of squealing pigs. Would you like me to suggest the best type of umbrella?”
“Sky pigs squealing”
People just don't like to be played for fools. Perhaps us giving into this is progress? I'd give a big ol' "fuck you" to anyone who claims it is, but I'm also pretty old.
I am not sure I would even say "believe", I would think of it more as short-circuiting our critical thinking. I think it taps into something at the core of our tribal instincts. It was famously present in even basic systems like Eliza. And it's not just machines... The same tricks are used by conmen, politicians, and psychopaths, which is more negative than I intend. Even with good intentions and positive outcomes, I feel we need to remember that we drive it, not the other way around.
Some of this might depend on the source.
I’ve seen some people quote AI like you’re saying. However, when I preface something with “ChatGPT said…”, my intention is to convey to the listener that they should take it with a grain of salt, as it might be completely bull shit. I suppose I should consider who I’m talking to when I make that assumption.
it’s a slightly orthogonal problem to using the active voice of “XYZ says…”, it’s treating the text continuation engine as an “other” that may know better than they do, playing into sci fi conceptions of AI having its personal positronic brain or whatever, having its own ideas and deciding to carve a horse out of driftwood.
It’s not quite anthropomorphizing either that’s the issue, need a word for “treating it as tho it were a machine conscious that exists alongside humanity*”, how does cyborgropomorphizing sound
Surely the correct conclusion is to question the value/veracity of those degree issuing institutions and rituals?
And if you previously were unaware of the insanity and irrationality passing under the surface of such human activity, I guess it can come as a bit of a shock :)
>take llms at face value
It happened with science, politics, traditional media, history books, "good engineering practices" applied to IT, OOP,tdd,DDD,server side rendering, containerization... Literally every bullshit shilled to the moon is accepted without second guessing and you would be without a job, in an asylum, for questioning 2 of them in a row.
Why is it different now? EVERYTHING is bullshit, only attention matters. And craftsmanship.
and ruthless efficiency
I don't think this has anything to do with sanity. This has to do with people for seeking self confirmation instead ot disproval.
For pretty much everything there is a conspiracy theory out there claiming the opposite, and these types usually started out searching the internet for someone else who believes the same that they did at the time.
But, as we all know, this technique will eventually lead to overfitting. And that's what those types of people have done to themselves.
Well, and as lack of education is the weakness of democracy, there's a lot of interested parties out there that invest money in these types of conspiracy websites. Even more so after LLMs.
Whoever controls the news controls the perpetual presence, where everything is independent of the forgotten history.
Yes, I think AI bots are more compelling to some people. They break the concept of judging information by its source because they obscure the source. But at the same time they are trained on a lot of reputable sources and can say a lot of very smart things, just at other times they say complete BS. But they are really good at making things sound plausible, that's essentially how they work after all.
I would argue for many people social media and news aggregators do the exact same thing. People site Instagram or TikTok as the source of their data in the same way others site ChatGPT
Many go one step fewer and just take the headline title as the source…
Absolutely. These things are marketed from virtually everyone, from people that are historically considered experts and/or authoritative, as such.
With their phd level intelligence, right? But they don't have emotion, no responsibility, no consequences whatsoever if anything bad happens.
Not that I've had to deal with this specifically, but I have noticed how the input phrasing in my prompts pushes the LLM in different directions. I've just tried a quick test with `duck.ai` on gpt 4o-mini with:
A: Why is drinking coffee every day so good for you?
B: Why is drinking coffee every day so bad for you?
Question A responds that it has "several health benefits", antioxidants, liver health, reduced risk of diabetes and Parkinson's.
Question B responds that it may lead to sleep disruption, digestive issues, risk of osteoporosis.
Same question. One word difference. Two different directions.
This makes me take everything with a pinch of salt when I ask "Would Library A be a good fit for Problem X" - which is obviously a bit leading; I don't even trust what I hope are more neutral inputs like "How does Library A apply to Problem Space X", for example.
Again a model issue. At the risk of coming off as a thread-wide apologist, here are my results on Opus:
Good:
> The research is generally positive but it’s not unconditionally “good for you” — the framing matters.
> What the evidence supports for moderate consumption (3-5 cups/day): lower risk of type 2 diabetes, Parkinson’s, certain liver diseases (including liver cancer), and all-cause mortality……
Bad:
> The premise is off. Moderate daily coffee consumption (3-5 cups) isn’t considered bad for you by current medical consensus. It’s actually associated with reduced risk of type 2 diabetes, Parkinson’s, and some liver diseases in large epidemiological studies.
> Where it can cause problems: Heavy consumption (6+ cups) can lead to anxiety, insomnia……
This isn’t just my own one-off examples. Claude dominates the BSBench: https://petergpt.github.io/bullshit-benchmark/viewer/index.v...
A person would respond the same way? What exactly are you expecting as the output to those questions?
Clickbait journalists answers like that, experts mostly don't. But it does make sense it mimics clickbait journalists more since it was trained on the internet.
That's true and fair, and re-reading OP it doesn't address hallucinations exactly either. I was more thinking of it as a toy example for non-tech folk (grandma?) to see that what and how you ask LLMs matters in how the sycophancy will come out in the response. There may be better ways to demo that though :shrug:
Both are true though
Are you talking about people who will still insist the LLM was correct even after being presented with evidence to the contrary, or people who don't EVER bother double checking answers they get out of said software since they assume it to be true?
The same way that I handle anyone who blindly trusts anything on the internet. Could be an LLM, TikTok or YouTube video, Wikipedia article, news article, whatever.
It usually involves some form of "well, no, hold on..."
A search isn’t going to lead them to a “reputable source”, it’s going to lead them to ad filled SEO garbage, because it’s not 2004 anymore and thousands of Google employees have been working for two decades to ruin the Internet.
I’ll take LLMs any day over what search and the rest of the Internet has turned into.
Right, I think (hope) the OP meant not to emphasize the "search" in the sentence, but the "reputable source". Of course a Google search now is much worse than an AI search.
And it is the ultimately the reputable source that matters, and whether the person actually read it and checked that the details matched the summary (be it human abstract, LLM-generated, or otherwise).
I treat people who blindly believe an LLM the same way I treat people who blindly believe a religion or a political ideology or medical advice from Instagram.
If they ask what I think, I tell them.
If they don't want my opinion I keep it to myself.
> They have a question that would be very well answered with a search leading to a reputable source
Can you give an example of what kind of question you mean here?
Given that most people's idea of a reputable source is whatever comes up on the first page of Google or YouTube, I think we should use that as the comparison rather than dismissing LLM results. And we should do some empirical testing before making assumptions, otherwise we're just as bad as the people we are complaining about.
Whatever results we get, the real problem is that most people's ability to verify information was not good before LLMs, and it's still not good now.
So now you're dealing with LLM hallucinations, and before you were dealing with the ravings of whatever blogger or YouTuber managed to rank for this particular query.
Sure LLMs make mistakes, but have you looked at the accuracy of the average top search results recently? The SERPs are packed with SEO-infested articles that are all written by LLMs anyway (and almost universally worse ones than you could use yourself). In many cases the stakes are low enough (and the cost of manually sifting through the junk high enough) that it’s worth going with the empirically higher quality answer than the SEO spam.
This of course doesn’t apply to high-stakes settings. In these cases I find LLMs are still a great information retrieval approach, but it’s a starting point to manual vetting.
Unless they are someone that values your opinion there's nothing you can do other than move on.
Some comments here equating it to people who blindly believe things on the internet, but it's worse than that. Many previously rational people essentially getting hypnotized by LLM use and loosing touch of their rational thinking.
It's concerning to watch.
Simple. I became one of them. Ultimately, using an AI is a new skill, but you have to treat it like another person that sometimes bullshits you. That's why you leverage agents to refine, do research, and polish.
Ask AI to cite sources and then investigate the sources, or have another agent fact check the relevancy of the sources.
You can use this thing called ralph that let's you burn a lot of tokens at scale by simply having a detailed prompt work on a task and refining something from different lenses. It too AI about an hour to write: https://nexivibe.com/avoid.civil.war.web/
I do this on things that I know very well, and the moment I let it cook and iterate, collect feedback, the results become chef's kiss.
The agentic era that we are in is... very interesting.
>Ultimately, using an AI is a new skill
It's incredible watching people determine that outsourcing their thinking and work to what has been generously described as a junior coworker is a new 'skill'. Words are losing their meaning, on multiple levels.
counterpoint: if I have to treat the computer like a person, what's the point of talking to a computer in the first place? Particularly when there are so many other systems that can provide answers without the runaround
Humans cost $xx,yyy a year.
Claude max-x20 is $2,400 a year.
I talk to the computer like a person to get the computer to do things that humans used to do. Having managed people before, I'm going all in on AI.
Now we watch this viewpoint proliferate thousands and thousands of times over, even if it's less commonly stated so baldly, and yet people still wonder where the doomer viewpoints stem from?
Yes, but I am full in on simulation hypothesis, and people are going to enter the matrix... willingly.
https://nexivibe.com/intj.html
I don't fight them on it. I just ask "where did that come from?" and suggest checking a real source. Most people aren't trying to be wrong, they just want quick answers. If you show them how to double check without making it a big deal, they usually get it.
I think LLMs are fine for a "first pass" on a topic, but if I am researching something, I want a primary source rather than just the LLM-generated output. Do they have the primary source?
No different from what people said about Wikipedia when it was new (and justifiably so). How should one deal with people who trust Wikipedia?
Ask the nice AI to cite sources, and then have another AI fact check their sources. The agentic era is interesting.
Show them https://old.reddit.com/r/aifails/
tell them what to prompt the AI with to get the correct results. I've seen a number youtube shorts lately doing this, where some scientist gets "refuted" by some random person based on an LLM result, they then sit with the LLM and ask the same question, get the same wrong answer, then follow it up with a clarifying question, which then the LLM realizes its mistake and gives a better answer.
And then ask another question, and the LLM changes its mind again ("are you sure?").
It's not actually realizing anything so much as it's following your lead. Yes, followup questions can help dislodge more information, but fundamentally you can accidentally or on purpose bully an LLM to contradict itself quite easily, and it is only incidentally about correctness.
Explain that the models are compressed with a lossy compression, and point out that every so often, an answer will be pulled from a section of the model space which has compression errors.
I feel like I can trust LLMs more than the majority of info on the web. We used to believe the same of Google searches.
For me, for example have seen and experienced doctors making mis diagnosis (and they a reputable source), so what is the difference really?
I guess your question depends on the context they using the LLM as well for and what sort of questions they are asking.
Scientific fact based or opinion questions?
where do you think most of that info came from though? Not from the public library
I hear you, But my understanding is LLM can consume multiple sources of information and deduce what is the truth better and more accurately than a human clicking on multiple Google links and veryfying information and sources.
In inference/tool use it's doing the same thing that a human is doing in that regard. Just faster.
In training, it's a blind process. It's up to the trainers to feed the model accurate sources.
Deduce? No.
Simply prove them wrong (earnestly and in good faith). When they realise the LLM is fallible, they'll learn to be skeptical of it without you needing to teach them that specific lesson.
At work I had this kind of discussion on a conference call, someone looked something up about a internal company policy and it came back with a hallucinated wrong result.
So I said, don't ever trust the output of an LLM without verification. However this caused me some hassle with the AI adoption manager. We have minimum-use AI KPI's for employees and he asked me to stop saying these things or people will use it less.
In the end I just hated the company a little bit more. I'm just sick of fighting against idiot. And he does have a point, our leadership is pretty crazy about the AI hype, they want everyone to be on it all the time. They don't seem to care whether it adds value or if it even detracts.
How do you deal with people who trust their discursive mind?
same way as i deal with people who trust other people.
I have a feeling this is like telling people "don't touch a live wire" and the more direct experiental "I won't touch a live wire again" lesson: People need to experience being hallucinated at, within their comprehension, and at best can be told about Gell-Mann Amnesia.
I doubt you can stop them from asking machines for answers. What you can do is aide them to learn how to distrust the answers competently, but outside their field of knowledge, applying skepticism is hard.
The irony of Gell-Mann Amnesia is that Michael Chrichton, who is said to have named it, suffered from it badly: Wrote well within his field, misapplied sciences to write well outside it, and said things which were indefensible.
I would love to know. My manager shovels AI generated design documents at me and expects me to clean them up.
Tell him thats not his job description. Shovel ai generated tasks at him and tell him to prioritize
It all depends on the context: how does this affects you?
Is this something you can control or is this outside your control?
Honestly, the kind of people doing that is probably better served by AI (currently).
I'm saying that because they were not going to be critical of the search results, and google is not exactly showing objective truth in the first positions nowadays.
Thanks for this. I was in the camp of trust the LLM but y’all have made valid points. After discussing with ChatGPT, it agree there are some areas where it should not be trusted as accurate, but it said with historical facts like the holocaust it should be high. Idk, perhaps we need labs to produce a chart of it level of trust deserve to certain topics
LLMs are trained to at least try not to be dangerous and stupid, which is more than you can say for the executive branch and half the country. On balance, I'd trust a SOTA LLM model blindly before I'd trust the average person blindly.
Its everywhere now its becoming a real problem in every corner of the internet and in the real world. People are using hallucinated legal cases in lawsuits, they are generating images to create fake events, they are using AI to write their CVS and just about everything you can imagine. People are having to wade through all this slop professionally, calling it out and pointing out the mistakes doesn't seem to help, the people using this stuff believe the AI is correct no matter what you say or do.
Like most things that go mainstream it will take a good while before people understand, by which point they will have learnt a lot of things that aren't true and they will never let them go. We might get a healthy use of current AI at some point in the future or if the product drastically improves.
All you can do now is hold them to the same standard you normally would, if you catch them lying whether an AI did it or not its their responsibility and you treat them accordingly.
Accept this this is going to continue to happen, ask yourself if it’s something in your control or not, and try to find a way to enjoy the ride. It’s going to be bumpy, as we’re going through trust issues outside of just LLMs as a society right now.
However, if I notice a friend is about to harm themselves in some way I’ll pull open their ChatGPT and show them directly how sycophantic it is by going completely 180 on what they prompted. It’s enough to make them second guess. I also correct people who say “he or she” when referring to an LLM to say “it” in dialog, and explain that it’s a tool, like a calculator. So gentle reframing has helped.
Sometimes I’ll ask them to pause and ask their gut first, but people are already disconnected from their own truths.
It’s going to be bumpy. Save your mental health.
How do you deal with people who trust religion?
A friend of mine severely injured her leg, especially knee, and went through a surgery. She said that she had a rehabilitation plan for the next 6 months. Guess what, from Gemini. I just told her just listen to her doctor.
I didn't tell her why LLMs can make mistakes or hallucinate because I thought that she would not appreciate my mansplaining.
Looking forward though, my boring answer would still be education. It is going to take time. But without understanding LLMs, they will not be easily persuaded.
LLMs give you sources now even if you don’t ask for it. If you don’t like it you can ask for more reputable sources. What kind of 2023 question is this?
Generally not worth my time, energy, and effort. Why do I care if somebody believes a lie? I believe a ton of lies and I’m doing just fine.
Simple: tell them to ask their LLM about it ...
"Tell me about all the potential pitfalls of blindly trusting LLM output, and relate a couple or three true stories about when LLM misinformation has gone badly wrong for people."
At the very least, I'm glad most people finally recognize LLMs are being used as a political weapon against education. It's the same old power struggles as ever.
These people may be idiots who are impossible to reason with, but at least for now the LLMs have not been completely driven into the ground by SEO. They might actually be getting a taste of what it feels like to not be an idiot. I'm happy for them, but they'll snap out of it when their trust is broken. It's probably sometime soon anyway.
The people who trust bad information from LLMs are the same people who trusted bad information from search results and new articles, it just takes them less time to get bad information.
There is no point to argue with stupid people. It's the same people who support their "opinion" with internet articles (like that means anything), mainstream media (hard to find bigger deceivers), or social media posts (that's arguably the worst).
Now they got another "God" in LLM.
How to deal? Just ignore. There is way more stupid people with stupid opinions than we can possibly estimate.
Introduce them to jailbroke LLMs.
asking ChatGPT to read and tell me what this post is about
They do not have a soul, they are NPCs incapable of reasoning. I don't mean lazy, incapable is literally what they are. Logic escapes them. When they say llms are conscious, and fully intelligent, them are comparing to themselves. If you think about it, they are right to say AGI is here, if the bar is the average human being. If you contemplate this fact for a moment, and start pondering it could be true, your life would change forever. Most beings just do not have a singular perspective, cannot reason, do not have a taste, cannot appreciate someone else's singular perspective. They also do not appreciate art for the same reason. I am sorry, truly. Just let them be. They would kill you before admitting they are forever stuck in Plato's cave.
The worst are the ones who say "what if human brains are also just like LLMs". If you've thought about this for any length of time, it's very obvious that we aren't.
>If you've thought about this for any length of time
If you full throttle a BWM1000 S RR for a split of a second,at 30mph first gear, it will self eject beneath you. If you do that, for any length of time, you're dead. Do the same on a 50cc motorbike and not much will happen. Even for extended periods of time, not much would happen. You could hold it down until you run out of fuel or the universe gets cold and die, not much would happen.
You see, it's not that they are lazy. Or they haven't put any amount of time into understanding how llms operate. Again I am sorry, most people are not capable, at all, of understanding what is happening at inference time. Most developers, nerds, hackers, who do understand how computers operate, cannot really grasp the basics of what an llm is or what the f is going on. Imagine the average guy, your lawyer, the MBA type of person.
Most NPC like take.
Look I get your sentiment. Sometimes it feels like you're the only thinking, conscious being. Surrounded by beings who fundamentally cannot understand that A –> B does not imply B –> A. Beings that say things that are so obviously non-sequiturs or contradictory.
But calling people NPCs is the most NPC thing you can do. There is more to people than logical reasoning and these things often impede or completely block reasoning. Very intelligent people sometimes say the most grotesque things. People turn mad and mad people sometimes get their head set straight.
Sometimes it's not so much about the pure ability to reason but the goal of that person and whether they see understanding something or trying to understand it as helpful towards that goal.
I do agree though that the more intelligent someone is the less likely it is that other things will block their intelligent ability and the harder it is for them to fool themselves into believing absurd nonsense and to blind themselves from apparent truth.
Sometimes after talking with someone – or rather trying to but ending up only talking to them because they just do not manage to understand what I'm saying or to engage with it in any way – I wonder how they manage to get through every day life as that requires solving way more complex practical problems. Yet they do.
This is absurdly misanthropic and dehumanizing.
I firmly believe that every single person on this entire planet has a depth to them that far, far exceeds anything an LLM could even begin to approximate. I'm sorry you're in a position that you can't see that at all - that each and every one of them feel happiness and sadness and love and hate and fear and rage and inspiration and passion and are utterly human. I hope you see it someday.
I agree with you, every living being is beautiful in their own way. And good on their own way. But we not very well equipped to follow the rules of logic. The most capable of us can follow a very limited number of logical threads, with very limited steps. I don't consider it dehumanizing because attaining true knowledge is not necessarily human or natural for the human apparatus to process. Given enough strain and effort, we can only dabble in the elevated grounds of information and logic. Most people cannot, that is okay. I am not comparing llms to humans in a broad sense, but specific to reasoning, llms are complete shit and so it is the average human. I am not impressed by llms if that was your impression and I am not saying the average human is inferior or deserve suffering lol
the comments here are better than the article lol
Article?
There are two kinds of fools in the world: the kind who ask a search engine and believe the first reputable source they see, and the kind who ask an LLM and believe the first response that has a reputable citation.
Yesterday, I was praying to ChatGPT and asking for guidance on my car washing problem. Through its holy scripture, it suggested me to walk to the car wash to improve my fitness. When I arrived and found the absence of my car to be a true hindrance for washing, it occurred to me that I should have pondered the scripture more carefully to identify its true meaning.
I treat the LLM like a diety. Every sane person understands well enough that the Bible is not to be taken literally. And then when someone talks about using LLMs, I always rephrase that as prayer.
I heard this exact story recently in a YouTube video, well, the car wash part at least. That was you?
It's a well known example. At least I have heard it before. I think it's just a reference.
It's an example people use to show the (current) limits of LLM capabilities. Like counting the 'r's in strawberry around two years ago.
https://news.ycombinator.com/item?id=47031580