It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.
I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.
Arbitration is an inefficient and unproductive process. I suppose that may be what the parties chose, but the public will likely never know what happened and the problem will be allowed to reoccur again and again. Things like these are better resolved in courts, with res judicata, precedent, and visibility so that legislators can fix statutes where necessary.
There's a reason we used to have courts with public records. Public records and transparency are a good thing. Binding arbitration has destroyed all of that.
I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down
Then you should be scared about every single online account you use, since they all have this same capability of suspension of your account. That's inherent in any service policy.
I don't suspect anyone here is demanding laws be written requiring every single player in any SaaS outage to make a public statement immediately following an event like this. That's an odd thing to state your preference on.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.
We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.
Yep. Google doing Google things. Not everyone stays and just keeps using them though, we're actively planning to remove GCP from our primary workloads now and will cut our spend to about 1/10 of current as we keep them in the stack as a cold multi-cloud failover target only.
Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite
As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down.
Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.
I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors?
it's not like they're the only provider, or even the #1
The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.
Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).
I can't believe that readers of HN actually think that that is how it does or should work.
Google/GCP can only make very general statements and in this case we want more than that.
They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.
Either way, we need to hear about this from Railway.
It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.
I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.
Arbitration is an inefficient and unproductive process. I suppose that may be what the parties chose, but the public will likely never know what happened and the problem will be allowed to reoccur again and again. Things like these are better resolved in courts, with res judicata, precedent, and visibility so that legislators can fix statutes where necessary.
There's a reason we used to have courts with public records. Public records and transparency are a good thing. Binding arbitration has destroyed all of that.
I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
Would you be okay with GCP making public issues you encounter with your account?
Well, I would host workloads on GCP... provided I could easily move them elsewhere and I just treat them as disposable.
What about Azure?
Do you care about uptime and security?
https://damrnelson.github.io/github-historical-uptime/
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
HN discussion: https://news.ycombinator.com/item?id=47616242
Honestly the best counterattack Iran could make rn would be somehow convincing the Pentagon to move everything to Azure
Google really should publish a flow diagram for how they decide to turn off someone's business.
Would you want your vendors publicly disclosing potentially private reasons for an outage?
With consent, yes.
Is there any indication Railway has consented to such disclosure?
But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?
Is that what you'd want your vendors to do?
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example.
If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
> if it wasn't something specific to their setup
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
a huge account like theirs should not be subject to automated actions like that.
an entire gcp project deleted along with its persistent disks.
how does that make any sense? nobody thought to call them or anything
This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down
Then you should be scared about every single online account you use, since they all have this same capability of suspension of your account. That's inherent in any service policy.
It depends on what you mean by ‘need’. If you mean they should for PR purposes, I probably agree.
If you are saying they should be required to by law, then no I disagree.
I don't suspect anyone here is demanding laws be written requiring every single player in any SaaS outage to make a public statement immediately following an event like this. That's an odd thing to state your preference on.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.
We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.
Yep. Google doing Google things. Not everyone stays and just keeps using them though, we're actively planning to remove GCP from our primary workloads now and will cut our spend to about 1/10 of current as we keep them in the stack as a cold multi-cloud failover target only.
Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- See https://imgur.com/a/x0Y3RJl
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.
I've directly asked our account manager about it. It's pretty scary that we don't know what automated mechanisms could just cut us off.
I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors?
it's not like they're the only provider, or even the #1
The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.
This is exactly why people get nervous about platform risk
Post your ask in the discussion: https://news.ycombinator.com/item?id=48204770
https://news.ycombinator.com/item?id=48201484
Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).
That statement is the last 15 or so years.
Railway can simply move to other service. We all know Google in unreliable, so why should google give public statement?
Thanks.