I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks than to burn it by causing a massive, visible problem.
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
T-Mobile had this issue 5 years or so ago where the IMS core went down due to fiber issues at one of their major backbone providers (pretty sure it was zayo).
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
>Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
That’s really weird that’s the case. DNS simply resolves “google.com” to an IP address (8.8.8.8 or something). Shouldn’t impact anything download related. I’m pretty sure DNS isn’t used for geolocating either
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
One of these times they will be right and you will never hear the end of the time they were first to recognize the start of a cyberattack.
You know what they say about a stopped clock: it's wrong 1,438 times a day.
If it is a quartz watch, it might be 47,120,384 times per day.
It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart
When telcos get compromised again, the attackers should just take it down in service of this moronic fallacy.
https://www.wsj.com/politics/national-security/china-cyberat...
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
In general, whats the expected cost of being loudly, obviously, publicly and obnoxiously wrong?
Keep in mind who the President is.
That's not true though, sometimes it's inexplicitly DNS too.
My money is on expired certificate.
If so, successful test.
The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
it's always DNS
or BGP.
I worked at a major ISP and we had a similar situation where the North East went down and the RC was a fiber cut at a major node in Philly.
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
The major Canadian outage was Rogers in 2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks than to burn it by causing a massive, visible problem.
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
In a dead empire, sufficiently advanced rot is indistinguishable from malice.
Infrastructure in general seems worse than 20 years ago. Our talent for black-bagging dictators has never been stronger, though!
It's affecting every mobile carrier (ATT, TMO), not just Verizon
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
Further info: https://old.reddit.com/r/cellmapper/comments/1qcwj48/observa...
T-Mobile had this issue 5 years or so ago where the IMS core went down due to fiber issues at one of their major backbone providers (pretty sure it was zayo).
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
Really neat concept but that name is so unfortunate. Is it bit-chat or bitch-at? Frankly either would work.
It's obviously bit chat
feelin bitch-at, frfr
Checking off "complains about the name" on my HN bingo card.
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
Is there a substantive connection?
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
>Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Telco infra runs on autopilot?
This is really taking the whole "I could build Dropbox in a weekend" style comments to a new level.
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
Carrier-level telco is labor intensive.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
VZW has 146m lines. Each employee supports 14,600 customers, seems like a reasonable number...
lol. You figured it out, Verizon wireless can just be replaced by a UniFi router and ChatGPT.
ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
DC local. Total outage this afternoon. SOS on my phone for a few hours. Unable to log into my account.
At 2:14PM EST, Verizon said:
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
Odds are they have a ton of outsourced engineers who are an ocean away and unable to do anything.
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
At least the shareholders have had an amazing quarter. :)
I am in Miami right now, and verizon is not working here. (I get the SOS sign).
The link just redirects me to https://www.youtube.com/c/FirstCoastNews.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
why are you saying that this is affecting more than one provider?
Some outlets reporting T-Mobile and ATT as well.
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
Estonia was the first major NATO victim of such things https://en.wikipedia.org/wiki/2007_cyberattacks_on_Estonia
Made worse by the fact Estonia is a more networked society than, for example, the US.
The alternative network reports are most likely people trying to call Verizon customers and reporting an outage when they can't get through.
You think like a person who’s debugged large systems failures before :). That feels very plausible.
The down detector site has Verizon outage reports two order of magnitude bigger, so it doesn't seem like a cyber attack to me.
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
T-Mobile is up and Verizon is down in my house
It’s Denmark! Ο_o
Here's a map: https://downdetector.com/status/verizon/map/
That's a population map
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
oh, so it's a heat map?
https://xkcd.com/1138/
Verizon acknowledged the issue: https://xcancel.com/VerizonNews/status/2011500483072954495
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
>Note, you have to click on the providers to see absolute magnitude
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
> Verizon acknowledged the issue: > https://xcancel.com/VerizonNews/status/2011500483072954495
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
Our Comcast DNS keeps going in and out too, not clear if it is related or knock on effect or something else entirely.
I’m somewhat surprised you’re still using your ISP’s DNS when there’s ton of better free or paid options.
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
That’s really weird that’s the case. DNS simply resolves “google.com” to an IP address (8.8.8.8 or something). Shouldn’t impact anything download related. I’m pretty sure DNS isn’t used for geolocating either
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
"You relied on a service you pay for to actually work? What are you, stupid?"
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
When that service actively spies on you, then yes, that is stupid...
That’s… weirdly a thing.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
Yes actually, I happen to save the built in GPS for occasions when the phone network is inaccessible.
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Might be. VZ does host several big DNS root nodes. But it could just be comcast too...
Seems like it's happening on the West Coast to
Not sure why this says East coast, I am also affected here in Seattle.
Weird. My Verizon is working fine in Seattle.
Same in Seattle. Turned of Wifi and the data connection still let me online.
downdetector is showing a lot of down mobile networks & services
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
The down detector site has Verizon outage reports two order of magnitude bigger, so it doesn't seem like a cyber attack to me. ~60-160k vs ~1.5-1.8k
This is what happens when you invest more in lobbying than the network
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
Same. Femtocell is online but no clients, phones on SOS mode.
I could see Verizon access points but just couldn't connect to any of them. I had no issue sending texts over wifi though.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
West coast too, down in LA at same time (12 PM EST / 9 AM PST). I think they’re way way way underselling the # affected (“thousands” lol)
good morning sirs
Grok 4.1 down too: https://status.x.ai/api-us-east-1/INC10b35980
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
Sadly this has happened before:
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.
Didn’t they just raid a reporters house. I wonder what service provider they used…