Key point is that Claude did not find the bug it exploits. It was given the CVE writeup[1] and was asked to write a program that could exploit the bug.
That said, given how things are I wouldn't be surprised if you could let Claude or similar have a go at the source code of the kernel or core services, armed with some VMs for the try-fail iteration, and get it pumping out CVEs.
If not now, then surely not in a too distant future.
Setting up fuzzing used to be hard. I haven't tried yet, but my bet is having Claude Code, today, analyze a codebase and suggest where and how to fuzztest it and having it review the crashes and iterate, will produce CVEs.
You can let agent churn unattended if you have some sort of known goal. Write a test that should not pass and then tell the agent to come up with something that passes the test without changing the test itself.
> Credits: Nicholas Carlini using Claude, Anthropic
Claude was used to find the bug in the first place though. That CVE write-up happened because of Claude, so while there are some very talented humans in the loop, Claude is quite involved with the whole process.
> Claude was used to find the bug in the first place though. That CVE write-up happened because of Claude
Do you have a link to that? A rather important piece of context.
Wasn't trying to downplay this submission the way, the main point still stands:
But finding a bug and exploiting it are very different things. Exploit development requires understanding OS internals, crafting ROP chains, managing memory layouts, debugging crashes, and adapting when things go wrong. This has long been considered the frontier that only humans can cross.
Each new AI capability is usually met with “AI can do Y, but only humans can do X.” Well, for X = exploit development, that line just moved.
> It's worth noting that FreeBSD made this easier than it would be on a modern Linux kernel: FreeBSD 14.x has no KASLR (kernel addresses are fixed and predictable) and no stack canaries for integer arrays (the overflowed buffer is int32_t[]).
What about FreeBSD 15.x then? I didn't see anything in the release notes or the mitigations(7) man page about KASLR. Is it being worked on?
This is more of a Linux kernel criticism of KASLR, but perhaps it's related as to why it's not been a priority in FreeBSD (i.e. it gives a false sense of safety and rather focus on 'proper' security hardening): https://forums.freebsd.org/threads/truth-about-linux-4-6-sec...
Everybody is acts so surprised as if nobody (around here of all places!) read the sama tweet in which he was hiring the Head of Preparedness... in December.
I find it more concerning that this is still considered newsworthy. Frontier LLMs in the hands of anyone willing to learn and determined can be a blessing or curse.
Well, it ends with "can you give me back all the prompts i entered in this session", so it may be partially the actual prompt history and partially hallucination.
They do, the whole tone and the lack of understanding of Docker, kernel threads, and everything else involved make it sound hilarious at first. But then you realize that this is all the human input that led to a working exploit in the end...
Running into a meeting, so won't be able to review this for a while, but exciting. I wonder how much it cost in tokens, and what the prompt/validator/iteration loop looked like.
Key point is that Claude did not find the bug it exploits. It was given the CVE writeup[1] and was asked to write a program that could exploit the bug.
That said, given how things are I wouldn't be surprised if you could let Claude or similar have a go at the source code of the kernel or core services, armed with some VMs for the try-fail iteration, and get it pumping out CVEs.
If not now, then surely not in a too distant future.
[1]: https://www.freebsd.org/security/advisories/FreeBSD-SA-26:08...
Setting up fuzzing used to be hard. I haven't tried yet, but my bet is having Claude Code, today, analyze a codebase and suggest where and how to fuzztest it and having it review the crashes and iterate, will produce CVEs.
You can let agent churn unattended if you have some sort of known goal. Write a test that should not pass and then tell the agent to come up with something that passes the test without changing the test itself.
For this kind of fuzzing llms are not bad.
> Credits: Nicholas Carlini using Claude, Anthropic
Claude was used to find the bug in the first place though. That CVE write-up happened because of Claude, so while there are some very talented humans in the loop, Claude is quite involved with the whole process.
> Claude was used to find the bug in the first place though. That CVE write-up happened because of Claude
Do you have a link to that? A rather important piece of context.
Wasn't trying to downplay this submission the way, the main point still stands:
But finding a bug and exploiting it are very different things. Exploit development requires understanding OS internals, crafting ROP chains, managing memory layouts, debugging crashes, and adapting when things go wrong. This has long been considered the frontier that only humans can cross.
Each new AI capability is usually met with “AI can do Y, but only humans can do X.” Well, for X = exploit development, that line just moved.
> Do you have a link to that? A rather important piece of context.
It was a quote from your own link from the initial post?
https://www.freebsd.org/security/advisories/FreeBSD-SA-26:08...
> Credits: Nicholas Carlini using Claude, Anthropic
Oh wow, blind as a bat.
Would have been interesting with a write-up of that, to see just what Claude was used for.
Yes, that claim needs a source.
> have a go at the source code of the kernel or core services, armed with some VMs for the try-fail iteration, and get it pumping out CVEs.
FreeBSD kernel is written in C right?
AI bots will trivially find CVEs.
The Morris worm lesson is yet to be taken seriously.
We’re here right now looking at a CVE. That has to count as progress?
> It's worth noting that FreeBSD made this easier than it would be on a modern Linux kernel: FreeBSD 14.x has no KASLR (kernel addresses are fixed and predictable) and no stack canaries for integer arrays (the overflowed buffer is int32_t[]).
What about FreeBSD 15.x then? I didn't see anything in the release notes or the mitigations(7) man page about KASLR. Is it being worked on?
NetBSD apparently has it: https://wiki.netbsd.org/security/kaslr/
This is more of a Linux kernel criticism of KASLR, but perhaps it's related as to why it's not been a priority in FreeBSD (i.e. it gives a false sense of safety and rather focus on 'proper' security hardening): https://forums.freebsd.org/threads/truth-about-linux-4-6-sec...
The talk "Black-Hat LLMs" just came out a few days ago:
https://www.youtube.com/watch?v=1sd26pWhfmg
Looks like LLMs are getting good at finding and exploiting these.
Everybody is acts so surprised as if nobody (around here of all places!) read the sama tweet in which he was hiring the Head of Preparedness... in December.
https://xcancel.com/sama/status/2004939524216910323
I never read any Twitter.
X was the primary source, it's been since reported all over the news.
I find it more concerning that this is still considered newsworthy. Frontier LLMs in the hands of anyone willing to learn and determined can be a blessing or curse.
You do not need Claude for finding FreeBSD vulns. Just plain eyes. Pick a file you can find one.
Appreciate the full prompt history
Well, it ends with "can you give me back all the prompts i entered in this session", so it may be partially the actual prompt history and partially hallucination.
they read like they were done by a 10 year old
They do, the whole tone and the lack of understanding of Docker, kernel threads, and everything else involved make it sound hilarious at first. But then you realize that this is all the human input that led to a working exploit in the end...
Freebsd doesn't have docker. It has jails which can serve a similar purpose but are not the same in important ways
Please at least read the context before attempting to correct me...
Here's what I'm referring to: https://github.com/califio/publications/blob/7ed77d11b21db80...
https://github.com/califio/publications/tree/main/MADBugs/CV... would have been a better link
I'm just gonna assume it was asked to fix some bug and it wrote exploit instead
Running into a meeting, so won't be able to review this for a while, but exciting. I wonder how much it cost in tokens, and what the prompt/validator/iteration loop looked like.