I still remember following Andries’s “Linux kernel hacker’s hut” course he taught at the Eindhoven University of Technology (TU/e) back in 2010. Every week we’d get an assignment where we had to write exploits for commonly occurring security vulnerabilities (e.g., buffer overflows, bad printf format). It was one of the most enjoyable courses I ever followed. Thanks for that, Andries!
I’d say, let the one who tried to allocate memory crash, and if you’re a critical process like xlock, use statically allocated memory and don’t alloc again.
The problem is that Linux has memory overcommit and it will OOM when a process faults a page in, not just when someone allocates memory.
So the OOM condition can hit any random process, not necessarily one that just tried to allocate. If you don't have some sort of selection, then you would still have an OOM killer, only it will be killing completely at random.
Statically allocated memory can still OOM on access, due to overcommit and lazy page table population. What you really want is mlockall(2) (probably with MCL_CURRENT|MCL_ONFAULT followed by madvise with MADV_POPULATE_*)
This is only a viable answer when overcommit is disabled. The problem comes when overcommit is enabled and you find yourself in a position where many programs think they already have memory and yet there is none to give them. If you simply kill the first piece of code that encounters the end of available memory you might take down anything including the kernel itself.
Nothing like statically allocating memory can work when overcommit is enabled because the kernel is free to compress memory, page it out and etc. and then murder you the next time you try to perform any operation that it doesn't have the space for, no matter how safe and static your initialization was.
Note that overcommit is very useful in many cases including the ones where swap saves the stability of the system under conditions that would otherwise completely lock up or panic, so it's also not viable to just prevent it from being used.
OOM killer always felt like a band-aid on a severed artery to me. I've rarely seen a machine that got into OOM state really recover without a full reboot.
IMO if the security of a system depends on the lock screen not crashing then the system is not very secure. Security protocols should never fail open like that; a lock screen should never simply be a layer on top of the authenticated desktop. Windows and macOS get this right. I believe Wayland display managers are also able to get this right (but I haven't checked).
I never pay for the OOF insurance, it seems like a waste of money and I've never met anyone that's had it happen.
I still remember following Andries’s “Linux kernel hacker’s hut” course he taught at the Eindhoven University of Technology (TU/e) back in 2010. Every week we’d get an assignment where we had to write exploits for commonly occurring security vulnerabilities (e.g., buffer overflows, bad printf format). It was one of the most enjoyable courses I ever followed. Thanks for that, Andries!
It's 2026 and I still can't configure the OOM killer to kill firefox before anything else.
I always wanted it to target java processes, as they were always the culprit. These days it's python, VSCode, and antigravity.
This. It's always browser running amok. I configured win+k shortcut key to: killall -9 chrome
I’d say, let the one who tried to allocate memory crash, and if you’re a critical process like xlock, use statically allocated memory and don’t alloc again.
> if you’re a critical process like xlock, use statically allocated memory and don’t alloc again.
This doesn't save you if someone other allocates and OOM killer chooses you as victim
What is proposed is to not have an OOM killer with a selection process, meaning that the "someone other allocates" would be the one dying.
The problem is that Linux has memory overcommit and it will OOM when a process faults a page in, not just when someone allocates memory.
So the OOM condition can hit any random process, not necessarily one that just tried to allocate. If you don't have some sort of selection, then you would still have an OOM killer, only it will be killing completely at random.
Yes, don’t have OOM roulette.
Statically allocated memory can still OOM on access, due to overcommit and lazy page table population. What you really want is mlockall(2) (probably with MCL_CURRENT|MCL_ONFAULT followed by madvise with MADV_POPULATE_*)
This is only a viable answer when overcommit is disabled. The problem comes when overcommit is enabled and you find yourself in a position where many programs think they already have memory and yet there is none to give them. If you simply kill the first piece of code that encounters the end of available memory you might take down anything including the kernel itself.
Nothing like statically allocating memory can work when overcommit is enabled because the kernel is free to compress memory, page it out and etc. and then murder you the next time you try to perform any operation that it doesn't have the space for, no matter how safe and static your initialization was.
Note that overcommit is very useful in many cases including the ones where swap saves the stability of the system under conditions that would otherwise completely lock up or panic, so it's also not viable to just prevent it from being used.
OOM killer always felt like a band-aid on a severed artery to me. I've rarely seen a machine that got into OOM state really recover without a full reboot.
I’m not against taking down the kernel if the situation is that catastrophic. Better than killing the lock screen for sure.
Shouldn't desktop environments detect if a lock screen terminated abnormaly anyway? The OOM killer is just one of many possible causes.
IMO if the security of a system depends on the lock screen not crashing then the system is not very secure. Security protocols should never fail open like that; a lock screen should never simply be a layer on top of the authenticated desktop. Windows and macOS get this right. I believe Wayland display managers are also able to get this right (but I haven't checked).
The fact that xlock crashing unlocks an X11 session is, IMO, pathetic.
looking forward to your other insights
(2004)
Thanks. I was confused for a bit, given these days you can do
to disable OOM killing for a process.https://github.com/torvalds/linux/blob/master/include/uapi/l...