In my previous company, we "simply" used fixed versions for our dependencies. And we had our own NPM registry that only had already approved packages for specific version. Approval required a security review by someone from the Security team… At first I was super annoyed by this. But I started to like this approach. It also reduced surprises while developing in a team… "it works on my machine" was rare since everyone was using the exact same versions. And moving to a newer version was done on a regular basis but it was an intentional thing we did.
I’ve mentioned this before, but at my previous employer we set up staged Artifactory so tha production couldn’t pull from anything that hadn’t been through the test stage, and test couldn’t pull from anything that hadn’t been through CI.
Because releases were relatively slow (weekly) compared to other places I worked (continuous), we had a reasonable lead time to have third party packages scanned for vulns before they made it to production.
The setup was very minimal, really just a script to link one stage’s artifacts to the next stage’s repo. But the end effect was production never pulled from the internet and never pulled packages that hadn’t been deployed to the previous stage.
>> But the end effect was production never pulled from the internet
Having production ever pull from the interwebs just seems bonkers to me. Even if you (for some reason?) want to stay up-to-date on every single dependency release you should be pulling to your own repo with some sort of gated workflow. If you're doing continuous deployment you definitely want to put extra control around your external dependencies, and releasing your product quickly after they change is probably the rare exception.
Something similar is easy with docker. Build the image when releasing to your first stage env, deploy the very same image to the next stage, until it reaches production. Nothing can break in between and enough time to test.
Reminds me of my favorite setup for a React project. We had rotating deployed builds (like three or more of them) for pull requests, the React UI pointed to a specific system so you always had consistent data, devs would immediately be able to test the UI, QA could follow immediately after.
What prevents the malicious piece of a package from just staying dormant for the cooldown period so nobody notices? What happens if everyone waits for a week to cool down, so the exact same problem happens, only a week later?
This may work in the case the maintainer becomes aware of the compromised account/credentials/package before the cooldown period, otherwise if it's about vendors and "other people", it's a roll of the dice. They may have longer cooldown periods than you.
Then we absolutely need an override for the cooldown period when we have to pull a zero-day patch the moment it's released.
The only reason cooldown periods haven't been exploited it's because they're not widely utilized. If they become mainstream it'll take a couple weeks to be rendered useless (yes this last part is 100% futurology)
I'm a little bit amused by the inclusion of .NET in this article.
The last time I pulled in more than Dapper I was using .NET Framework 4.8. Batteries are very included now. Perhaps a cooldown on dapper and maybe two other things would protect me to some degree, but when you have 3rd party dependencies you can literally count on one hand, it's hard to lose track of this stuff over time. I'd notice any dependency upgrade like a flashing neon sign because it happens so rarely. It's a high signal event. I've got a lot of time to audit them when they occur.
I think there's a lot more than you might initially realize. A few of the top of my head (beyond your ORM): automapper, 3rd-part json or Polly, logging, server-side validation, many more. Another vector: unlikely a lot of other languages I've found way more .net libraries for 3rd party connectors or systems are community-based.
.NET definitely includes more these days, including lots of the things I've mentioned above, but they're often not as good and you likely have legacy dependencies.
Bun added `trustedDependencies` [1] to package.json and only executes postInstall scripts coming from these dependencies. I think this is something that should be supported across all JS package managers, even more than version cooldowns.
How can you know that a dependency you trust won't be hacked? At best it slightly reduces the risk, but it's not even close to the effectiveness of version cooldowns that just block 100% of fresh updates.
It's a fine balancing act between getting the latest updates and avoiding supply chain attacks.
I completely understand the author here, because I'm actually also leaning more towards avoiding supply chain attacks than jumping on the latest CVEs.
It's just a gut feeling, rooted in 25 years of experience as a sysadmin, but I feel like a supply chain attack can do a lot more damage in general than most unpatched known vulnerabilities.
Just based on my own personal experiences, no real data.
I'll try to put words to it, but a supply chain attack is more focused, higher chance of infilitration. While a CVE very rarely is exploited en masse, and exploitation often comes with many caveats.
That combined with the current state of the world, where supply chain attacks seem to be a very high profile target for state actors.
and those rare zero-days can be treated as the exception, and dealt with quickly. It seems backwards to optimize for dependency change reaction time these days with the supply chain such an attractive target.
This is security through obscurity (which is to say: it's not security). Malware already waits dormant for years in many different attack vectors. There are known, simple fixes for the attacks on package managers. They need to implement those fixes, not ineffective hacks.
> Malware already waits dormant for years in many different attack vectors
And some malware doesn't wait. Sure, some supply chain attacks like the one in Notepad++ are much more sophisticated, but some untargeted ones (like the recent Cline CLI one) rely on package managers doing thousands of downloads before it's noticed and stopped.
- Could you explain what you mean by "security through obscurity"? The mechanism is well explained in the blog.yossarian.net posts linked within. It is simply adding a time filter on a client.
- Also, I'm not sure if package registries (e.g. server) and package managers (e.g. client) are being conflated here regarding "attacks on package managers", this seems to be more of a mitigation a client could do when the upstream content in a registry is compromised.
- Lastly, I agree with the sentiment that this is not a full solution. But I think it can be useful nevertheless, a la Swiss Cheese Safety Model. [1]
Has any notable package manager tried the staggered rollout model that approximately every one of us uses when deploying changes at our day jobs? The complexity is, of course, version compatibility/mismatch, but it seems like a solvable problem. Then you could have an automatic canary system.
The closest I've seen to this are opt-in early release channels.
I think the premise is that modern scanners are really good at finding malicious code (and are run by dozens of companies in the industry), but when it gets pushed and installed inside of that 7 day window, the spread is uncontrolled. This basically gives you opportunity to let the machinery in the package ecosystem do it's job.
If everyone only starts using a package after 7d, it feels like it just means we don't find out about problematic packages until 7d later . The reason 7d works is because it is the "don't go first" effect (I assert with no evidence). But if everyone does the same delay, there's no longer a benefit to you delaying. This feels like a prisoners dilemma of no one upgrading.
I do think there is some sense in having some cool down. Automated review systems having some time to sound alarms would be good.
I'm not sure what the reporting mechanisms look like for various ecosystems. Being able to declare that there should be a hold is Serious Business, and going through with a hold or removal is a very human-costly decision to make for repo maintainers. With signficiant lag. So we are up to at least 2d, if centralized.
Ideally I'd like to see something on atprotocol, where individuals can create records on their PDS's that declare dangers. This can form a reputation system, that disincentivizes bad actors (false reports), and which can let anyone on the net quickly see incoming dangers, in a distributed fashion, real time.
IIUC the recent high-profile npm backdoors were mostly detected by supply-chain-security firms that ingest all package updates from the registry and look for suspicious code using automated or semi-automated analysis. Dependency cooldowns work great with this kind of thing. I agree that, if malicious packages were mostly detected via user reports, dependency cooldowns would create a prisoners' dilemma.
I don't understand what you're saying about reporting mechanisms; is there something wrong with how this is currently done?
Maybe a better way would be to allow third-parties to certify releases, and you can specify only to pull the package once they've given it the green light.
I've been working on a project lately as my bachelor's dissertation which I later plan on working on long term on this issue.
The basic premise is a secure package registry as an alternative to NPM/PyPi/etc where we use a bunch of different methods to try to minimize risk. So e.g. reproducible builds, tracing execution and finding behavioral differences between release and source, historical behavioral anomalies, behavioral differences with baseline safe package, etc. And then rather than having to install any client side software, just do a `npm config set registry https://reg.example.com/api/packages/secure/npm/`
eBPF traces of high level behavior like network requests & file accesses should catch the most basic mass supply chain attacks like Shai Hulud. The more difficult one is xz-utils style attacks where it's a subtle backdoor. That requires tests that we can run reproducibly across versions & tracing exact behavior.
Hopefully by automating as much as possible, we can make this generally accessible rather than expensive enterprise-only like most security products (really annoys me). Still definitely need a layer of human reviews for anything it flags though since a false positive might as well be defamation.
Won't know if this is the right direction until things are done & we can benchmark against actual case studies, but at least one startup accelerator is interested in funding.
It is too bad pkgsrc from NetBSD did not become a thing, a few Linux distros allow for its use and I think Minux also uses it.
Using pkgin(1) you get to know what happens before anything is done. Creating packages is hard, but as an end user pkgsrc shows me what it will do before it does anything.
This example for installing a binary package is from an already active NetBSD workstation, all items needed will be shown for the package you want to install.
# pkgin install gnumeric-1.12.59nb2
calculating dependencies...done.
4 packages to install:
gnumeric-1.12.59nb2 goffice0.10-0.10.59nb2 lasem-0.6.0nb1 libgsf-1.14.54
0 to remove, 0 to refresh, 0 to upgrade, 4 to install
16M to download, 76M of additional disk space will be used
proceed ? [Y/n] n
This is theater. I say this every time it comes up, but the solution here IS NOT to just add vain and silly friction between potentially-malicious upstreams and package users. You'll never win that war.
The solution is independent audit. "Package managers" need to be (as they are in the Linux world) human beings responsible for integrating, validating and testing the upstream software for the benefit of their users.
NPM, PyPI, Cargo et. al. continue to think they can short circuit that process and still ship safe software, and they verifiably cannot.
This is silly. Critical security and bug fixes come out and you are going to wait because you think older must be safer just to avoid a supply chain attack issue? Just secure the supply chain. Be critical about your dependencies and update strategy before updating. If you got some 200 translative dependencies and you don't know everything your build, that is a problem and you probably should look into that because assuming waiting is a solution is not going to stop you from getting hurt with that risk of a surface area.
In the age of AI, I reduced my load on small utility libraries and just have the bigger ones that I'll follow semver and update to manager versions when it make sense and always take small patches but still look at the release notes for what changed.
Surprised not to see nix mentioned in connection with this topic!
If you use nix (especially nix flakes), this consideration falls out naturally from the nixpkgs repository reference (SHA, branch, etc) you choose to track. Nixpkgs has various branches for various appetites for the "cutting edge".
Yes! Nixpkgs straddles both worlds, like a system package manager it provides a way to install packages and their dependencies. However, like most language package managers it also imposes a locking mechanism so that every input into the nix expression* is locked to a hash, the mechanism is recursive and handles multiple versions of the same package in parallel.
The recent(ish) concept of "nix flakes" means there are 2 related but different mechanisms for achieving this but the end result is the same.
* In the land of NixOS everything is a nix expression including the system packages, configuration files, install image. It's all locked to hashes of upstream sources and is in theory, fully byte-identical reproducible.
Yeah, definitely. Think of it as what the post calls a "system" package manager. The difference between nix and the SPMs mentioned by the post is that in the former case, control over dependencies lies with you, not with the package manager.
In other words, with nix you decide the spec of the software you want installed on your machine, not the maintainer of your chosen package manager. Depending on your use case and knowledge/experience level, either choice may be preferable.
Also, nixpkgs is definitively the widest-spanning "package manager" on the "market"; see link.
In my previous company, we "simply" used fixed versions for our dependencies. And we had our own NPM registry that only had already approved packages for specific version. Approval required a security review by someone from the Security team… At first I was super annoyed by this. But I started to like this approach. It also reduced surprises while developing in a team… "it works on my machine" was rare since everyone was using the exact same versions. And moving to a newer version was done on a regular basis but it was an intentional thing we did.
I’ve mentioned this before, but at my previous employer we set up staged Artifactory so tha production couldn’t pull from anything that hadn’t been through the test stage, and test couldn’t pull from anything that hadn’t been through CI.
Because releases were relatively slow (weekly) compared to other places I worked (continuous), we had a reasonable lead time to have third party packages scanned for vulns before they made it to production.
The setup was very minimal, really just a script to link one stage’s artifacts to the next stage’s repo. But the end effect was production never pulled from the internet and never pulled packages that hadn’t been deployed to the previous stage.
>> But the end effect was production never pulled from the internet
Having production ever pull from the interwebs just seems bonkers to me. Even if you (for some reason?) want to stay up-to-date on every single dependency release you should be pulling to your own repo with some sort of gated workflow. If you're doing continuous deployment you definitely want to put extra control around your external dependencies, and releasing your product quickly after they change is probably the rare exception.
Something similar is easy with docker. Build the image when releasing to your first stage env, deploy the very same image to the next stage, until it reaches production. Nothing can break in between and enough time to test.
Reminds me of my favorite setup for a React project. We had rotating deployed builds (like three or more of them) for pull requests, the React UI pointed to a specific system so you always had consistent data, devs would immediately be able to test the UI, QA could follow immediately after.
What prevents the malicious piece of a package from just staying dormant for the cooldown period so nobody notices? What happens if everyone waits for a week to cool down, so the exact same problem happens, only a week later?
This may work in the case the maintainer becomes aware of the compromised account/credentials/package before the cooldown period, otherwise if it's about vendors and "other people", it's a roll of the dice. They may have longer cooldown periods than you.
Then we absolutely need an override for the cooldown period when we have to pull a zero-day patch the moment it's released.
The only reason cooldown periods haven't been exploited it's because they're not widely utilized. If they become mainstream it'll take a couple weeks to be rendered useless (yes this last part is 100% futurology)
I'm a little bit amused by the inclusion of .NET in this article.
The last time I pulled in more than Dapper I was using .NET Framework 4.8. Batteries are very included now. Perhaps a cooldown on dapper and maybe two other things would protect me to some degree, but when you have 3rd party dependencies you can literally count on one hand, it's hard to lose track of this stuff over time. I'd notice any dependency upgrade like a flashing neon sign because it happens so rarely. It's a high signal event. I've got a lot of time to audit them when they occur.
I think there's a lot more than you might initially realize. A few of the top of my head (beyond your ORM): automapper, 3rd-part json or Polly, logging, server-side validation, many more. Another vector: unlikely a lot of other languages I've found way more .net libraries for 3rd party connectors or systems are community-based.
.NET definitely includes more these days, including lots of the things I've mentioned above, but they're often not as good and you likely have legacy dependencies.
Bun added `trustedDependencies` [1] to package.json and only executes postInstall scripts coming from these dependencies. I think this is something that should be supported across all JS package managers, even more than version cooldowns.
[1]: https://bun.com/docs/guides/install/trusted
That's security theater. The package can still run arbitrary code the moment it's actually used.
That could probably be solved by opting in to the permission model of Node. But that won't work for everybody, especially in legacy applications.
Having trusted dependencies at least drastically reduces the risk that 'git clone && npm install' takes over the entire system.
Cooling down dependencies would certainly help, also.
How can you know that a dependency you trust won't be hacked? At best it slightly reduces the risk, but it's not even close to the effectiveness of version cooldowns that just block 100% of fresh updates.
Can you help me understand why one would ever need a post-install script in the first place, please?
Ime the most reasonable case is an optional compilation of native components when prebuilt ones are not compatible. See also node-gyp
Some tools also install pre-commit hooks. I don't like this practice, but I get why people are using it.
To restart a service, or run ldconfig?
It's a fine balancing act between getting the latest updates and avoiding supply chain attacks.
I completely understand the author here, because I'm actually also leaning more towards avoiding supply chain attacks than jumping on the latest CVEs.
It's just a gut feeling, rooted in 25 years of experience as a sysadmin, but I feel like a supply chain attack can do a lot more damage in general than most unpatched known vulnerabilities.
Just based on my own personal experiences, no real data.
I'll try to put words to it, but a supply chain attack is more focused, higher chance of infilitration. While a CVE very rarely is exploited en masse, and exploitation often comes with many caveats.
That combined with the current state of the world, where supply chain attacks seem to be a very high profile target for state actors.
and those rare zero-days can be treated as the exception, and dealt with quickly. It seems backwards to optimize for dependency change reaction time these days with the supply chain such an attractive target.
This is security through obscurity (which is to say: it's not security). Malware already waits dormant for years in many different attack vectors. There are known, simple fixes for the attacks on package managers. They need to implement those fixes, not ineffective hacks.
> Malware already waits dormant for years in many different attack vectors
And some malware doesn't wait. Sure, some supply chain attacks like the one in Notepad++ are much more sophisticated, but some untargeted ones (like the recent Cline CLI one) rely on package managers doing thousands of downloads before it's noticed and stopped.
A few points/qs:
- Could you explain what you mean by "security through obscurity"? The mechanism is well explained in the blog.yossarian.net posts linked within. It is simply adding a time filter on a client.
- Also, I'm not sure if package registries (e.g. server) and package managers (e.g. client) are being conflated here regarding "attacks on package managers", this seems to be more of a mitigation a client could do when the upstream content in a registry is compromised.
- Lastly, I agree with the sentiment that this is not a full solution. But I think it can be useful nevertheless, a la Swiss Cheese Safety Model. [1]
[1]https://en.wikipedia.org/wiki/Swiss_cheese_model
Has any notable package manager tried the staggered rollout model that approximately every one of us uses when deploying changes at our day jobs? The complexity is, of course, version compatibility/mismatch, but it seems like a solvable problem. Then you could have an automatic canary system.
The closest I've seen to this are opt-in early release channels.
Does the timer start at upload, or does the timer start at mass exploitation?
Not a bad idea, but we'd need to have evidence of the former case to make it mandatory and widespread.
IIRC, many times it's compromised credentials of maintainers, which are caught very quickly, so that's evidence of the former case.
I think the premise is that modern scanners are really good at finding malicious code (and are run by dozens of companies in the industry), but when it gets pushed and installed inside of that 7 day window, the spread is uncontrolled. This basically gives you opportunity to let the machinery in the package ecosystem do it's job.
If everyone only starts using a package after 7d, it feels like it just means we don't find out about problematic packages until 7d later . The reason 7d works is because it is the "don't go first" effect (I assert with no evidence). But if everyone does the same delay, there's no longer a benefit to you delaying. This feels like a prisoners dilemma of no one upgrading.
I do think there is some sense in having some cool down. Automated review systems having some time to sound alarms would be good.
I'm not sure what the reporting mechanisms look like for various ecosystems. Being able to declare that there should be a hold is Serious Business, and going through with a hold or removal is a very human-costly decision to make for repo maintainers. With signficiant lag. So we are up to at least 2d, if centralized.
Ideally I'd like to see something on atprotocol, where individuals can create records on their PDS's that declare dangers. This can form a reputation system, that disincentivizes bad actors (false reports), and which can let anyone on the net quickly see incoming dangers, in a distributed fashion, real time.
(Hire me, I'll build it.)
IIUC the recent high-profile npm backdoors were mostly detected by supply-chain-security firms that ingest all package updates from the registry and look for suspicious code using automated or semi-automated analysis. Dependency cooldowns work great with this kind of thing. I agree that, if malicious packages were mostly detected via user reports, dependency cooldowns would create a prisoners' dilemma.
I don't understand what you're saying about reporting mechanisms; is there something wrong with how this is currently done?
Maybe a better way would be to allow third-parties to certify releases, and you can specify only to pull the package once they've given it the green light.
This seems like it would cause the https://xkcd.com/989/ effect.
I've been working on a project lately as my bachelor's dissertation which I later plan on working on long term on this issue.
The basic premise is a secure package registry as an alternative to NPM/PyPi/etc where we use a bunch of different methods to try to minimize risk. So e.g. reproducible builds, tracing execution and finding behavioral differences between release and source, historical behavioral anomalies, behavioral differences with baseline safe package, etc. And then rather than having to install any client side software, just do a `npm config set registry https://reg.example.com/api/packages/secure/npm/`
eBPF traces of high level behavior like network requests & file accesses should catch the most basic mass supply chain attacks like Shai Hulud. The more difficult one is xz-utils style attacks where it's a subtle backdoor. That requires tests that we can run reproducibly across versions & tracing exact behavior.
Hopefully by automating as much as possible, we can make this generally accessible rather than expensive enterprise-only like most security products (really annoys me). Still definitely need a layer of human reviews for anything it flags though since a false positive might as well be defamation.
Won't know if this is the right direction until things are done & we can benchmark against actual case studies, but at least one startup accelerator is interested in funding.
It is too bad pkgsrc from NetBSD did not become a thing, a few Linux distros allow for its use and I think Minux also uses it.
Using pkgin(1) you get to know what happens before anything is done. Creating packages is hard, but as an end user pkgsrc shows me what it will do before it does anything.
This example for installing a binary package is from an already active NetBSD workstation, all items needed will be shown for the package you want to install.
This is theater. I say this every time it comes up, but the solution here IS NOT to just add vain and silly friction between potentially-malicious upstreams and package users. You'll never win that war.
The solution is independent audit. "Package managers" need to be (as they are in the Linux world) human beings responsible for integrating, validating and testing the upstream software for the benefit of their users.
NPM, PyPI, Cargo et. al. continue to think they can short circuit that process and still ship safe software, and they verifiably cannot.
Oh the lengths we go to avoid the obvious solution out of laziness...
This is silly. Critical security and bug fixes come out and you are going to wait because you think older must be safer just to avoid a supply chain attack issue? Just secure the supply chain. Be critical about your dependencies and update strategy before updating. If you got some 200 translative dependencies and you don't know everything your build, that is a problem and you probably should look into that because assuming waiting is a solution is not going to stop you from getting hurt with that risk of a surface area.
In the age of AI, I reduced my load on small utility libraries and just have the bigger ones that I'll follow semver and update to manager versions when it make sense and always take small patches but still look at the release notes for what changed.
Surprised not to see nix mentioned in connection with this topic!
If you use nix (especially nix flakes), this consideration falls out naturally from the nixpkgs repository reference (SHA, branch, etc) you choose to track. Nixpkgs has various branches for various appetites for the "cutting edge".
Can you explain how this works? Is it different from what's described in the "Language vs. system package managers" section of the post?
Yes! Nixpkgs straddles both worlds, like a system package manager it provides a way to install packages and their dependencies. However, like most language package managers it also imposes a locking mechanism so that every input into the nix expression* is locked to a hash, the mechanism is recursive and handles multiple versions of the same package in parallel.
The recent(ish) concept of "nix flakes" means there are 2 related but different mechanisms for achieving this but the end result is the same.
* In the land of NixOS everything is a nix expression including the system packages, configuration files, install image. It's all locked to hashes of upstream sources and is in theory, fully byte-identical reproducible.
Yeah, definitely. Think of it as what the post calls a "system" package manager. The difference between nix and the SPMs mentioned by the post is that in the former case, control over dependencies lies with you, not with the package manager.
In other words, with nix you decide the spec of the software you want installed on your machine, not the maintainer of your chosen package manager. Depending on your use case and knowledge/experience level, either choice may be preferable.
Also, nixpkgs is definitively the widest-spanning "package manager" on the "market"; see link.
https://en.wikipedia.org/wiki/Nix_(package_manager)#Nixpkgs