Sensitive does not mean it is not readable. It is just simply not exposed through the UI. It can be easily leaked if you return a bit too much props from the action functions or routes.
The only way to defend against these types of issues is to encrypt your environment with your own keys, with secrets possibly baked into source as there are no other facilities to separate them. An attacker would need to not only read the environments but also download the compiled functions and find the decryption keys.
It is not ideal but it could work as a workaround.
please don't suggest this. The right way is to have the creds fetched from a vault, which is programmed to release the creds auth-free to your VM (with machine level identify managed by the parent platform)
This is just another layer of indirection (which isn't bad; it adds to the difficulty of executing a breach). The fundamental problem with encrypted secrets is that at some point you need to access and decrypt them.
"Effective defense requires architectural change: treating OAuth apps as third‑party vendors, eliminating long‑lived platform secrets, and designing for the assumption of provider‑side compromise."
Designing for provider-side compromise is very hard because that's the whole point of trust...
As someone trying to think about OAuth apps at our SaaS, it certainly is very hard.
Do any marketplaces have a good approach here? I know Cloudflare, after their similar Salesloft issue, has proposed proxying all 3rd party OAuth and API traffic through them. But that feels a little bit like trading one threat vector for another.
Other than standard good practices like narrow scopes, shorter expirations, maybe OAuth Client secret rotation, etc, I'm not sure what else can be done. Maybe allowlisting IP addresses that the requests associated with a given client can come from?
This was probably partly a Google refresh token theft (given the length of the access). No inside info, just looking at how the attack occurred.
OAuth 2.1[0] (an RFC that has been around longer than I've been at my employer) recommends some protections around refresh tokens, either making them sender constrained (tied to the client application by public/private key cryptography) or one-time use with revocation if it is used multiple times.
This is recommended for public clients, but I think makes sense for all clients.
The first option is more difficult to implement, but is similar to the IP address solution you suggest. More robust though.
The second option would have made this attack more difficult because the refresh token held by the legit client, context.ai, would have stopped working, presumably triggering someone to look into why and wonder if the tokens had been stolen.
Corroborates that zero-trust until now has been largely marketing gibberish. Security by design means incorporating concepts such as these to not assume that your upstream providers will not be utterly owned in a supply chain attack.
What bites people: rotating a vercel env variable doesn't invalidate old deployments, because previous deploys keep running with the old credential until you redeploy or delete them. So if you rotated your keys after the bulletin but didn't redeploy everything, then the compromised value is still live.
Also worth checking your Google Workspace OAuth authorizations. Admin Console > Security > API Controls > Third-party app access. Guarantee there are apps in there you authorized for a demo two years ago that are still sitting with full email/drive access.
Usually rotating a credential means that you invalidate the previous one. Never heard of rotating credentials that would only create new ones and keep the old ones active.
To me the biggest (but not only) issue is that blindly connecting sensitive tools to 3rd party services has been normalized. Every time I hear the word "claw" I cringe...
I still don't get how this exactly worked. Is the OAuth Token they talk about the one that you get when a user uses "Sign in with Google"? Aren't they then bound to the client id and client secret of that specific Google App the user signed in to? How were the attackers able to go from that to a control plane? Because even if the attacker knows the users OAuth token, the client id and the client secret, they can access the Google Drive etc. (which is bad, I get that) but I simply do not understand how they could log in into any Vercel systems from that point. Did they find the credentials in the google drive?
Once you have a session token, which is what you get after you complete the oauth dance, you can issue requests to the API. It is simple as that. The minted token had permission to access the victim's inbox, most likely, which the attacker leveraged to read email and obtain one-time passwords, magic links and other forms of juicy information.
Interesting - I wonder if this isn't a case of theft on a refresh token that was minted by a non-confidential 3LO flow w/PKCE. That would explain how a leaked refresh token could then be used to obtain access, but does the Vercel A/S not implement any refresh token reuse detection? i.e.: you see the same R/T more than once, you nuke the entire session b/c it's assumed the R/T was compromised.
Security-by-obfuscation is ridiculed but I'm a firm believer that preventing yourself from getting owned when someone is able to type 3 letters `env` is a worthy layer of defense. Even if those same secrets are unencrypted somewhere else on the same system, at least make them spend a bunch of time crawling through files and such.
It's ridiculed because its no protection on its own when an attacker is motivated. Its fine to add as an additional layer though if you want to make your space mildly custom to protect against broader attacks.
I don't see how its necessarily relevant to this attack though. These guys were storing creds in clear and assuming actors within their network were "safe", weren't they?
TFA cites "env var enumeration", likely implying someone got somewhere they shouldn't and typed 3 characters, as the critical attack that led to customers getting compromised.
My point is sensitive secrets should literally never be exported into the process environment, they should be pulled directly into application memory from a file or secrets manager.
It would still be a bad compromise either way, but you have a fighting chance of limiting the blast radius if you aren't serving secrets to attackers on an env platter, which could be the first three characters they type once establishing access.
I don't think that's what the attacker did here. Vercel is a PaaS product where other developers run apps. The enumerated environment variables were the env vars of Vercel's customers, which Vercel likely stores in a long-term data store. Rather than running `env` on a Linux box somewhere, the attacker may have just accessed that data store.
I'm not sure I've seen it mentioned yet that when Vercel rolled out their environment variable UI, there was no "sensitive" option https://github.com/vercel/vercel/discussions/4558#discussion.... There was ~2 years or more until it was introduced https://vercel.com/changelog/sensitive-environment-variables...
Sensitive does not mean it is not readable. It is just simply not exposed through the UI. It can be easily leaked if you return a bit too much props from the action functions or routes.
The only way to defend against these types of issues is to encrypt your environment with your own keys, with secrets possibly baked into source as there are no other facilities to separate them. An attacker would need to not only read the environments but also download the compiled functions and find the decryption keys.
It is not ideal but it could work as a workaround.
> with secrets possibly baked into source
please don't suggest this. The right way is to have the creds fetched from a vault, which is programmed to release the creds auth-free to your VM (with machine level identify managed by the parent platform)
This is how Google Secrets or AWS Vaults work.
This is just another layer of indirection (which isn't bad; it adds to the difficulty of executing a breach). The fundamental problem with encrypted secrets is that at some point you need to access and decrypt them.
> OAuth trust relationship cascaded into a platform-wide exposure
> The CEO publicly attributed the attacker's unusual velocity to AI
> questions about detection-to-disclosure latency in platform breaches
Typical! The main failures in my mind are:
1. A user account with far too much privileges - possible many others like them
2. No or limited 2FA or any form of ZeroTrust architecture
3. Bad cyber security hygiene
Blaming AI is gonna be the security breach equivalent to blaming ddos when your website breaks isn't it.
That part of his tweet made me laugh out loud. I don't understand who it's directed toward.
"Effective defense requires architectural change: treating OAuth apps as third‑party vendors, eliminating long‑lived platform secrets, and designing for the assumption of provider‑side compromise."
Designing for provider-side compromise is very hard because that's the whole point of trust...
As someone trying to think about OAuth apps at our SaaS, it certainly is very hard.
Do any marketplaces have a good approach here? I know Cloudflare, after their similar Salesloft issue, has proposed proxying all 3rd party OAuth and API traffic through them. But that feels a little bit like trading one threat vector for another.
Other than standard good practices like narrow scopes, shorter expirations, maybe OAuth Client secret rotation, etc, I'm not sure what else can be done. Maybe allowlisting IP addresses that the requests associated with a given client can come from?
This was probably partly a Google refresh token theft (given the length of the access). No inside info, just looking at how the attack occurred.
OAuth 2.1[0] (an RFC that has been around longer than I've been at my employer) recommends some protections around refresh tokens, either making them sender constrained (tied to the client application by public/private key cryptography) or one-time use with revocation if it is used multiple times.
This is recommended for public clients, but I think makes sense for all clients.
The first option is more difficult to implement, but is similar to the IP address solution you suggest. More robust though.
The second option would have made this attack more difficult because the refresh token held by the legit client, context.ai, would have stopped working, presumably triggering someone to look into why and wonder if the tokens had been stolen.
0: https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1
I mean the admin account had visibility of clients env vars, thats maybe not really great in the first place.
Corroborates that zero-trust until now has been largely marketing gibberish. Security by design means incorporating concepts such as these to not assume that your upstream providers will not be utterly owned in a supply chain attack.
This article is solely overly wordy (probably ai) restatements of essentially just what vercel has publicly disclosed already
What bites people: rotating a vercel env variable doesn't invalidate old deployments, because previous deploys keep running with the old credential until you redeploy or delete them. So if you rotated your keys after the bulletin but didn't redeploy everything, then the compromised value is still live.
Also worth checking your Google Workspace OAuth authorizations. Admin Console > Security > API Controls > Third-party app access. Guarantee there are apps in there you authorized for a demo two years ago that are still sitting with full email/drive access.
Usually rotating a credential means that you invalidate the previous one. Never heard of rotating credentials that would only create new ones and keep the old ones active.
But then every rotation would break production, wouldn't it ?
When you rotate them, you supposed expire your old vars
yeah not redeploying on credential changes seems like a design flaw. Render redeploys on env var changes, for instance.
To me the biggest (but not only) issue is that blindly connecting sensitive tools to 3rd party services has been normalized. Every time I hear the word "claw" I cringe...
I still don't get how this exactly worked. Is the OAuth Token they talk about the one that you get when a user uses "Sign in with Google"? Aren't they then bound to the client id and client secret of that specific Google App the user signed in to? How were the attackers able to go from that to a control plane? Because even if the attacker knows the users OAuth token, the client id and the client secret, they can access the Google Drive etc. (which is bad, I get that) but I simply do not understand how they could log in into any Vercel systems from that point. Did they find the credentials in the google drive?
Once you have a session token, which is what you get after you complete the oauth dance, you can issue requests to the API. It is simple as that. The minted token had permission to access the victim's inbox, most likely, which the attacker leveraged to read email and obtain one-time passwords, magic links and other forms of juicy information.
Interesting - I wonder if this isn't a case of theft on a refresh token that was minted by a non-confidential 3LO flow w/PKCE. That would explain how a leaked refresh token could then be used to obtain access, but does the Vercel A/S not implement any refresh token reuse detection? i.e.: you see the same R/T more than once, you nuke the entire session b/c it's assumed the R/T was compromised.
I recently went to BreachForums and the space was filled with this
Do any services use vercel?
It's a really common platform for vibe coded sites, as I understand it.
Small startups often use it but typically outgrow it quickly unless they remain small and simple.
First of all, it is often used in Korea.
Security-by-obfuscation is ridiculed but I'm a firm believer that preventing yourself from getting owned when someone is able to type 3 letters `env` is a worthy layer of defense. Even if those same secrets are unencrypted somewhere else on the same system, at least make them spend a bunch of time crawling through files and such.
It's ridiculed because its no protection on its own when an attacker is motivated. Its fine to add as an additional layer though if you want to make your space mildly custom to protect against broader attacks.
I don't see how its necessarily relevant to this attack though. These guys were storing creds in clear and assuming actors within their network were "safe", weren't they?
TFA cites "env var enumeration", likely implying someone got somewhere they shouldn't and typed 3 characters, as the critical attack that led to customers getting compromised.
My point is sensitive secrets should literally never be exported into the process environment, they should be pulled directly into application memory from a file or secrets manager.
It would still be a bad compromise either way, but you have a fighting chance of limiting the blast radius if you aren't serving secrets to attackers on an env platter, which could be the first three characters they type once establishing access.
I don't think that's what the attacker did here. Vercel is a PaaS product where other developers run apps. The enumerated environment variables were the env vars of Vercel's customers, which Vercel likely stores in a long-term data store. Rather than running `env` on a Linux box somewhere, the attacker may have just accessed that data store.