> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice.
I'm not sure "worked properly" and "as intended" accurately describe this situation.
I also can't believe the people who were involved with writing this response from Meta, didn't realize how obviously bad it sounds. It's like there is no humans working and writing there anymore.
To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug.
In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.
> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address
> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address
It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.
I get the joke, but it's a relevant nuance that the new code, the chatbot, did not have 'the bug'. I still think that the mistake and head that should roll should be the one that published the chatbot.
But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.
The Password Verification System worked perfectly, as designed, yet there happened to be what we call a "bug" in the program that used it: the e-mail-address of the account owner was not properly associated with the login address, meaning that anyone with a valid password for any account could access all others with the same password.
That sounds a lot like the justifications Claude and ChatGPT give when confronted about something they did wrong, or when asked to provide a customer support response about software issues
I've lost track of the number of times Claude has basically said "it was like that when i got here" in the face of a clearly bogus choice and easily disproved explanation.
The argument here is that the AI is a glorified input page. The input field asks for your username and email and sends it to a backend function. Such an input page is working as intended.
The problem is when the backend function doesn't verify that the email matches the username.
Why on earth would the backend function even take an email?
Or perhaps said different: use the submitted info to identify the account; send any sensitive messages (recovery codes, password resets whatever) to only the contact info on file. If the chat bot can send such email it should do so via an API that sends only to contact info on file for the associated account and not to an email that's provided by the bot.
> Why on earth would the backend function even take an email?
In principle, it could be designed to do so to handle cases where a new email address has been confirmed out of band, e.g. for an account representing a company or a political office. But that's a relatively unusual situation, not something you'd want to be available to every user writing in. (Even if you had an all-human support department, this sort of functionality would only be available to a select few agents.)
I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.
> But when humans handled it, this was not as much as a problem.
In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers.
Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.
so how long was the bug there? was there a way to access it before/without the support agent? it feels like Meta will throw anything under the bus to redirect blame from the AI, because that would be the end of their $600B (depending on “which number you want to go with”) experiment
What I gather is that this internal tool was used by human support agents, and it was their responsibility to verify the email adresses and general validity of a claim.
But when implementing AGI TM that was overseen, maybe the oversight in the separate code path was a 'bug', but the mistake was making the chatbot obviously, if the separate code path had a bug, then it had become ossified into a feature, and it was internal, not exposed to the public.
This is an external communication, to save face sure, but if this is the internal excuse, it would be absolutely the wrong RCA and it reads as if the one who made the mistake is not admitting they made their mistake. Which to be honest, just making the mistake is enough to get fired, but not admitting it is enough to get ultra fired.
Did you create the account separately? Or as an asset of your main Meta account (like Meta Business Suite)?
I'm creating the accounts in Meta Business Suite, so I would have a recourse with my main personal account which can be linked to some adspend, so I'm assuming it will have better support channels than accounts created through an end-user interface.
This was probably my mistake. This is the first time I've ever done this so I just set up an account like a regular user. I didn't figure out how to link it with my Meta Business Suite until it was too late.
...They really ahouldn't have, and I wonder how this will affect all the big AI IPOs. After all, Meta is one of the big players in the space. Surely if they can't do it right, then...
Why was 'can a user request a different email' not literally the first test that comes to mind when making something like this? Do they not test anything because the scale is too big?
The nature of the invention is for people to relieve themselves of the burden of having to use their minds. And while there will be exceptions, (including, I'm sure you: the person reading this comment,) the vast majority of people are hungry to use AI in that spirit of being able to be lazy.
Because software professionals are conflating simplicity of user experience with simplicity of dev experience.
During development they were likely not thinking of the user experience, nor even the support agent experience, but on their development experience, they asked the LLM to develop the chatbot, and it worked, and the speed was documented and reported upstream so that shareholders invest, if there is any forethought it would go against the narrative of AI becoming the engineer or 100xing productivity.
I got a suspicious password reset request email today from Meta but it landed in my inbox. Luckily I have MFA and after checking audit logs inside IG upon logging in, I did not see anything suspicious.
If this was a bank that had zero humans and the AI chatbot was abused to hand over sensitive information about their customers which led to this disaster, people would never trust their bank ever again and leave.
Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop.
Applying a technical solution to a social problem almost always ends in disasters like this.
I’m guessing they have no functional human support for the people who had their accounts stolen. I get the impression Meta didn’t know this was happening until they were contacted by the media.
> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice.
I'm not sure "worked properly" and "as intended" accurately describe this situation.
In italian we say "l'operazione è riuscita perfettamente, ma il paziente è morto" -> "the surgery was a complete success, but the patient died"
Both this and what Meta said reminds me of "Clarke and Dawe - The Front Fell Off" (https://www.youtube.com/watch?v=3m5qxZm_JqM)
I also can't believe the people who were involved with writing this response from Meta, didn't realize how obviously bad it sounds. It's like there is no humans working and writing there anymore.
I was reminded of the Murray Walker quote. “There's nothing wrong with the car except it's on fire”
"operation successful, patient dead."
The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.
To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug.
[1]: https://www.documentcloud.org/documents/28202858-meta-ai-ag-...
In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.
> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address
> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address
Sounds like they are saying the agent did not malfunction, and this vuln could have been triggered by a human support agent too.
It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.
I get the joke, but it's a relevant nuance that the new code, the chatbot, did not have 'the bug'. I still think that the mistake and head that should roll should be the one that published the chatbot.
But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.
The Password Verification System worked perfectly, as designed, yet there happened to be what we call a "bug" in the program that used it: the e-mail-address of the account owner was not properly associated with the login address, meaning that anyone with a valid password for any account could access all others with the same password.
That sounds a lot like the justifications Claude and ChatGPT give when confronted about something they did wrong, or when asked to provide a customer support response about software issues
I've lost track of the number of times Claude has basically said "it was like that when i got here" in the face of a clearly bogus choice and easily disproved explanation.
Oh it was a downstream dependency. The tool worked, it was the downstream dependency. Glory to Arstotszka
The argument here is that the AI is a glorified input page. The input field asks for your username and email and sends it to a backend function. Such an input page is working as intended.
The problem is when the backend function doesn't verify that the email matches the username.
Why on earth would the backend function even take an email?
Or perhaps said different: use the submitted info to identify the account; send any sensitive messages (recovery codes, password resets whatever) to only the contact info on file. If the chat bot can send such email it should do so via an API that sends only to contact info on file for the associated account and not to an email that's provided by the bot.
> Why on earth would the backend function even take an email?
In principle, it could be designed to do so to handle cases where a new email address has been confirmed out of band, e.g. for an account representing a company or a political office. But that's a relatively unusual situation, not something you'd want to be available to every user writing in. (Even if you had an all-human support department, this sort of functionality would only be available to a select few agents.)
Fair enough. Never trust client-submitted browser form, but always trust LLM-submitted form.
I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.
>deterministic verification like this is not, and should never be, the LLM’s job.
But when humans handled it, this was not as much as a problem. That is, the humans did the job, because they recognized the need to do that job.
Sure sometimes accounts could get recovered if a human was tricked, but evidently it was easier to trick the LLM in masse than humans.
> But when humans handled it, this was not as much as a problem.
In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers.
Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.
Maybe they’re communicating exactly what it sounds like and are just owning up to being complete morons?
so how long was the bug there? was there a way to access it before/without the support agent? it feels like Meta will throw anything under the bus to redirect blame from the AI, because that would be the end of their $600B (depending on “which number you want to go with”) experiment
Then ‘ The tool itself’ was not appropriate to the job in the first place
How very Wernher von Braun of them.
Of course.
What I gather is that this internal tool was used by human support agents, and it was their responsibility to verify the email adresses and general validity of a claim.
But when implementing AGI TM that was overseen, maybe the oversight in the separate code path was a 'bug', but the mistake was making the chatbot obviously, if the separate code path had a bug, then it had become ossified into a feature, and it was internal, not exposed to the public.
This is an external communication, to save face sure, but if this is the internal excuse, it would be absolutely the wrong RCA and it reads as if the one who made the mistake is not admitting they made their mistake. Which to be honest, just making the mistake is enough to get fired, but not admitting it is enough to get ultra fired.
By "abusing" they mean "using"
Meanwhile an account I created for a new product was permanently disabled by an automated system with no path for me to appeal to a human.
(If anyone at Meta/Instagram sees this I wrote a brief blog post with the details. Please help! https://addisonwebb.com/blog/2026-06-05-Can%20Someone%20at%2... )
Did you create the account separately? Or as an asset of your main Meta account (like Meta Business Suite)?
I'm creating the accounts in Meta Business Suite, so I would have a recourse with my main personal account which can be linked to some adspend, so I'm assuming it will have better support channels than accounts created through an end-user interface.
This was probably my mistake. This is the first time I've ever done this so I just set up an account like a regular user. I didn't figure out how to link it with my Meta Business Suite until it was too late.
>AI-assisted account recovery system
oh no...Meta what are you doing
That sweet koolaid taste, how could one resist?
...They really ahouldn't have, and I wonder how this will affect all the big AI IPOs. After all, Meta is one of the big players in the space. Surely if they can't do it right, then...
This was on hacker news a few days ago (https://news.ycombinator.com/item?id=48359102) - description of the “hack”, not the cockamamie confirmation by Meta.
I really hope this accelerates meta's decline. The world will adapt just fine without social media.
Why was 'can a user request a different email' not literally the first test that comes to mind when making something like this? Do they not test anything because the scale is too big?
The nature of the invention is for people to relieve themselves of the burden of having to use their minds. And while there will be exceptions, (including, I'm sure you: the person reading this comment,) the vast majority of people are hungry to use AI in that spirit of being able to be lazy.
Because software professionals are conflating simplicity of user experience with simplicity of dev experience.
During development they were likely not thinking of the user experience, nor even the support agent experience, but on their development experience, they asked the LLM to develop the chatbot, and it worked, and the speed was documented and reported upstream so that shareholders invest, if there is any forethought it would go against the narrative of AI becoming the engineer or 100xing productivity.
In their defense, they asked the LLM to make no mistakes
I got a suspicious password reset request email today from Meta but it landed in my inbox. Luckily I have MFA and after checking audit logs inside IG upon logging in, I did not see anything suspicious.
2FA did not help according to the primary finding
https://news.ycombinator.com/item?id=48359102
Corrected headline: "Meta confirms 1000s of Instagram accounts were hacked due to their insecure AI chatbot".
And who said cameras linked to Meta in their glasses were a good idea?
"abusing" by using it's built in insecurity to do insecure things.
It's like, people abusing an open door. "Guys, just because we left the door open to your bedroom doesn't mean we're responsible".
God can only hope this is a business ending lawsuit.
It won’t be.
also this is more like them leaving the keys in the door, then someone comes along, uses the keys, and steals all your stuff.
truthfully, no equipment is actually defective in this scenario eh?
Probably some product manager pushed back on security considerations raised by engineers.
If this was a bank that had zero humans and the AI chatbot was abused to hand over sensitive information about their customers which led to this disaster, people would never trust their bank ever again and leave.
Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop.
Applying a technical solution to a social problem almost always ends in disasters like this.
Reputation can’t be vibe-coded.
Meta's brand is already toxic. Idk if there's much to lose there.
https://www.documentcloud.org/documents/28202858-meta-ai-ag-...
https://www.maine.gov/agviewer/content/ag/985235c7-cb95-4be2...
> Date(s) Breach Occured: 04/17/2026
> Date Breach Discovered: 05-31-2026
I’m guessing they have no functional human support for the people who had their accounts stolen. I get the impression Meta didn’t know this was happening until they were contacted by the media.