The comments here surprise me a bit. The common thread so far seems to be a general fear of US based companies, but how is that relates to the article?
Cloudflare's post is pretty boring here in that regard. They dig into how BGP works and propose that similar leaks seem common for the Venezuelan ISP in question.
Sure they could be wrong or even actively hiding the truth of what happened here, but the article mentions nothing of Cloudflare being involved in the action and they're describing a networking standard by pointing to publicly available BGP log data.
What am I missing here that everyone else seemed to zero in on?
I don't think this article provides any evidence of anything to be scared of.
That said, based on what we know already, there is no reason to take everything is this article at face value necessarily.
Firstly, if anybody isn't aware of the history of Stuxnet, it's worth reading, because otherwise you'd underestimate the government's ability to use 0-days by an order of magnitude (we're talking full custom-written multi-month hacking projects with root-kits and custom fake drivers delivered successfully to an airgapped system, source wikipedia). Also worth learning about Dual EC DRBG debacle.
Secondly am immediate friend of mine worked at a FANG company that routinely sent a firehose of all sorts of things matching all sorts of filters directly to governments. In fact many ISPS have back-doors built in and that's not really disputed (wikipedia: room641A).
So the question to ask yourself is -- if this was a deliberate interaction that cloudfare was required to participate in via a warrant, would they legally even be allowed to publish a blog post that contradicted this?
So I think that is probably the default attitude of skepticism you are seeing, which in my opinion is a good default. Plus the primary claim of this article "Look it wasn't 1 routing issue, it's been happening for even longer! Therefore nothing to look at here!" seems really weak.
> So the question to ask yourself is -- if this was a deliberate interaction that cloudfare was required to participate in via a warrant, would they legally even be allowed to publish a blog post that contradicted this?
So you're proposing they could be in a situation where they can either:
1. Publish an untruthful blog post, relying on public data available from multiple parties, trying to somehow explain it all while avoiding talking about their involvement in a way that would get them in PR, legal or political hot water; or
2. Publish nothing.
And they chose #1?
The only way #1 makes any sense at all is if some greater consequence to not publishing was put in place. But that would be more something like "the US gov essentially forced Cloudflare to write this" than "Cloudflare was part of this".
Unless they were part of this, _and_ the government forced them to write a post saying they're _not_ part of it and...
For my money: this is something in the news making it a good marketing opportunity which is ultimately what the blog is--trying to market Cloudflare and the brand to technical crowds.
For me number 1 is difficult basically because of who runs Cloudflare. I trust Matthew Prince because I find him to be: consistent and credible.
I work in go to market, specifically for businesses like Cloudflare, I can and have said "this real world situation is going to have resonance for the next 5-10 days, what is the lowest cost blog post you could publish that is related?" - because I only manage teams who produce content that is genuinely, at some level, value add or interesting to my target market, you would end up with a blog post exactly like this. In fact, this blog post is doing that job, here we are, cloudflare users, discussing cloudflare.
It becomes nuanced doesn't it? First thing is: to trust him fully is to understand what it means to trust him... that he knows his business well enough that he can intuitively feel things are wrong. That comes from not being checked out, so: he knows who is in his company and why, he knows the types of projects happening in his business and why, he has easy levers to gain real time information when something feels wrong, and - he monitors his business correctly. I trust Matthew because I know him, so I believe all those things are true. The final part is that trust is also about knowing that mistakes happen, and that they are being: sought out, addressed and owned. So when I say I trust him, it's because I believe everything aforementioned - it makes your scenario safe, at least to me.
> "Look it wasn't 1 routing issue, it's been happening for even longer! Therefore nothing to look at here!" seems really weak.
It's actually really strong since it implies that there's no real time-based correlation with the recent action in Caracas. Especially as the purported correlation was rather weak to begin with.
It's even older than Stuxnet, but either Dish Network (Echostar) or DirectTV did something similar in the early 2000's/late 90's.
They were having a lot of trouble with pirate receivers, so they added small chunks of code to normal device updates and this went on over a period of weeks/months. On the final update, it stitched all those bits of code together and every receiver that wasn't a legitimate one displayed the message "GAME OVER" on the screen and stopped working.
Obvs it was a long time ago so forgive me if I get some details wrong.
I looked at this a couple days ago and my thoughts were basically the same as Cloudflare's. It looks like a misconfiguration - one that's easy to make and isn't terribly uncommon. I can't rule out it wasn't an attack, but absent some other evidence, I don't see any reason to believe it was one.
That said, looking at their Cloudflare radar page now for AS8048, I don't recall there being any other BGP route leaks listed there for December from AS8048 and I definitely don't recall there being any BGP origin hijacks listed. The latter is something rather different from a route leak - that looks like someone blackholing some of CANTV's IPs.
I don't think I somehow just missed that since I definitely looked at CANTV's historical behavior to see if anything they did was unusual and that would have been one of the first things I checked, but perhaps they updated radar with data from other collectors or re-ran anomaly detection on historical data.
Ah yes, and we're back into "but my buddy told me " if you have to say that then your story just isn't worth saying or hearing and you should reconsider how impervious you are to conspiratorial thinking
The one thing they relied on "my buddy told me" for is actually not really in dispute as they say. Between CALEA, the Snowden leaks, and the earlier stuff (like the beamsplitters in Room 641A), we have known clearly based on a number of public and verifiable sources that the US government has its fingers deeply into the data streams that flow through US companies. This is a reasonable inference even absent all of this information.
Now ... I don't think any of this actually supports the parent comment's implication that Cloudflare took some anti-Venezuela action at the request of the US government, just that your criticism is kinda unfounded.
I share your view - how does this article imply US companies and/or government involvement? If there were such involvement what aspect of BGP gives the US entities more ability to carry this out vs other nefarious actors? I ask this sincerely knowing almost nothing about BGP and wanting to learn...
You may have missed https://news.ycombinator.com/item?id=46504963 a few days ago where this same anomaly was discussed and American government involvement was directly implied by the article.
The top comment of that thread points out exactly the same thing this Cloudflare article does; that there doesn't really seem to be be any indication this was anything nefarious.
Probably because most people only read headlines (and maybe 3 paragraphs) combined with the fact that the US has a long history of doing what people are condemning them for, even if this particular instance probably wasn't a case of such behavior. Especially considering how the general sentiment towards the US has gotten bitter with constant threads of invasion of Denmark and Canada by their government.
Or it's just Russian and China socket accounts? Who knows...
There was another post a few days ago that suggested a connection between the American invasion of Venezuela and the BGP anomaly: https://loworbitsecurity.com/radar/radar16/
Personally, I don't think the Americans would bother hide their attack and make it look like an accident under the current regime. Trump would announce the CIA/NSA/FBI/whatever did the Greatest Attack, and Amazing Attack, to Completely Control and break the Weak Government of Venezuela to Rescue Their Oil. I'll believe the "it was just a misconfiguration" explanation for now.
I think it only makes sense that people start fearing the influence of American companies given the current developments. When America is in the news, it's either threatening someone, pulling out of cooperative efforts, or delivering on a previous threat. That's bound to derail discussions whenever American companies are involved and it'll only get worse with the way things are developing.
That's what I find interesting about the billionaire elite standing behind el presidente, like, sooner or later he'll be gone and you guys -and your companies- won't. There's been no more compelling argument to actually overtax the rich to give to the masses than the last 13 months.
I'm half sleepy but I liked the post. The analysis regarding path prepending really drives the accident theory home. If a state actor were trying to intercept traffic (MITM), the last thing they would do is pad the AS path multiple times because that tells the global routing table, "Don't come this way, I am the long scenic route" lol
This could be a classic fat finger config error, most likely a route map intended to manipulate traffic engineering for their own upstream links that inadvertently leaked widely because of a missing deny-all clause. Neverthless, a good reminder that BGP is still fundamentally a trust based system where a single typo in a config file can cascade globally. Never attribute to malice that which is adequately explained by a missing export filter.
> If a state actor were trying to intercept traffic (MITM), the last thing they would do is pad the AS path
That's presumptuous: A state actor would (and could trivially) pad the wrong directions to flow traffic down to pops that are not making new announcements (and thus not-implicated by cloudflare and other "journalistic" efforts).
There's also a lot between fat-fingers and deep-state: I know of some non-state actors who do this sort of thing just to fuck with ad impressions. I also doubt much usable intelligence can be gained from mere route-manipulation thing, but I do know that if it is a fat-finger, every techdude in the area was busy at that time trying to figure it out, and wasn't doing their best work twelve hours later...
> most likely a route map intended to manipulate traffic engineering for their own upstream links
...that being said, this does seem plausible: Most smaller multihomed sites I've seen (and a few big ones!) have some kind of adhoc health monitoring/rebalance function that snmp or something and does autoexpect/curl or something-else to the router to run some (probably broken) script, because even if your uplinks are symmetrical, the rest of the Internet isn't, so route-stuffing remains the best way to manipulate ingress traffic.
> Never attribute to malice that which is adequately explained by a missing export filter.
As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them. Once I have a third, I have a legitimate need to manipulate my own ingress.
The problems with the BGP are legion, and not just one thing that prevents BGP and security from sharing time in a sentence.
> A state actor would (and could trivially) pad the wrong directions
This isn't how BGP works. An AS-PATH isn't the path the traffic will follow; it's the path that this overall announcement has allegedly tranversed and is (one of many attributes) used to judge the quality of route. The next hop tells our peer where they should send the data if they like this route.
Putting more things in the AS path makes the route less attractive. Leaking a new route isn't going to magically make some other route become more preferred.
Lower cost usually means lower quality and is an example of how a long path being leaked can result in traffic flowing away from high quality path to the leaked path.
Not saying that this is the case with Venezuela, just explaining the reality of BGP where path prepends are often ignored.
> > > That's presumptuous: A state actor would (and could trivially) pad the wrong directions to flow traffic down to pops that are not making new announcements
> > Leaking a new route isn't going to magically make some other route become more preferred.
> Not magic, but technology can look like magic when you don't understand it.
Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
I've used BGP for over 25 years, so I'm really curious what you're thinking. Or if you're describing something else, you're being really unclear.
Or if you're just describing withdrawing a route and replacing it with a really undesirable route -- sure, we do that all the time. But that doesn't match this scenario and isn't going to get flagged as a routing anomaly.
You know what's really toxic? Not explaining what you mean and just sending some introductory lab documentation about what the other person has already clearly shown they understand.
I don't even know what you mean by a lot of these things.. e.g.
> > > As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them.
A straightforward reading of "forward" doesn't work for this sentence. I should not take a route from peer A and send it to peer B. Peering isn't transitive. If I try, it should be filtered.
Peering means to give your own routes (and your transit customers' routes) to someone else. Not your other peers routes.
> Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
> ... I'm really curious what you're thinking
That the actor actually wanted the traffic to flow over route C.
> You know what's really toxic? Not explaining what you mean and just sending some introductory lab documentation about what the other person has already clearly shown they understand.
I think perhaps you and I have different ideas of what is "clear", for example when you said something that is totally covered in introductory lab documentation, I thought it was clear that you did not understand.
> I don't even know what you mean by a lot of these things
That is clear! But confusing! How can you clearly understand but not know what I mean?
> Peering means to give your own routes (and your transit customers' routes) to someone else.
That's exactly what's happening here: Not every transit customer peers with every other transit customer.
> > Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
Yes, but how does advertising undesirable route B make traffic go over route C? This is why I think you're confused.
> That's exactly what's happening here: Not every transit customer peers with every other transit customer.
I am not understanding what you're saying at all. You said:
> > > > As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them.
This is the thing you are supposed to never do as a peer, and the thing that I have a whole bunch of filtering to prevent my peers from inadvertently doing.
Are you misusing the word "peer"? It's hard to talk about BGP and routing policy without using these words correctly.
> I am not understanding what you're saying at all.
And that is why; You seem to have a very strong opinion about something that you don't understand "at all" and frankly I cannot understand how that can work.
> This is the thing you are supposed to never do as a peer
So you say, but that's what I did when back in the early 2000s, and that's what the parties in the news were doing, and if you're not totally lying to me, you know this because it's the default in BGP, that's why you would say you need to:
> I have a whole bunch of filtering to prevent my peers from inadvertently doing.
because that's how BGP works. Duh.
> It's hard to talk about BGP without using these words correctly.
and I am flabbergasted you continue to persist at it, when I have even offered you "introductory lab documentation" to help.
Peering means "give our downstream customers' routes plus our own routes; receive the same from them".
Transit means "give our entire table, receive their routes plus their downstream customers routes".
You don't give one peer's routes to another. You filter to make sure you are not doing this. They hopefully filter (using data from RIRs) to make sure you're not doing it. If both parties screw up the filtering, you "leak routes" like we're discussing here.
This has been standard practice for peering since at least 1997. It is codified, among other places, in RFC7454.
> And that is why; You seem to have a very strong opinion about something that you don't understand "at all" and frankly I cannot understand how that can work.
Do you operate an AS? Are you a peering contact? I mean, I only do it mostly for funsies now but for quite awhile that was part of my job. :P
Also, still seeking an answer to this question:
> > > Yes, but how does advertising undesirable route B make traffic go over route C [that previously went over route A]? This is why I think you're confused.
I set up multihoming in the US (going through ARIN assignment for ASN and PI) in the early 2000s and for another larger company in the UK (doing the same same but different) in the early 2010s.
> Also, still seeking an answer to this question:
Not sure what to tell you. I've answered this within the context of the news article, if you're asking specifically what kinds of configurations do that they're the kinds that are in that "introductory lab documentation" and if you're not overstating your credentials you should be able to understand.
I don't see the relation to BGP anomalies, since this "layer 3 shaping" is basically just "if you send traffic to the IP of an AS router, it probably goes over the link of that IP". None of this would help NSA "shape" arbitrary traffic onto links they are able to tap. (I'm really not sure what exactly the point of this is, the slides talk about exfil a lot, it would seem to me like some random device sending traffic to a router is more suspicious, because normal traffic never targets routers, than hitting an actual server somewhere but idk)
In en-us education "101" is often used to refer to an introductory course in a particular topic. My inference from the fact that this _educational_ slide is called "101" is that this is a basic example of core knowledge that people in this area of work are expected to have. It therefore stands to reason that there exists a "102" or "103" course that expands upon it, as well as material going far beyond "the syllabus".
The NSA and thirteen eyes generally have detailed traffic logging capability at core internet exchanges around the world. It is reasonable to think that a good way of exfiltrating data would be by having something like an ICMP or maybe even TTL based covert channel, such that there is no chance that the sent data is ever received by the recipient. I am just speculating – but that's why I thought this was interesting.
Funny to see even the NSA makes the mistake of calling a network an ASN (maybe because it's their name backwards), which is like saying I deposited money in my IBAN, or my neighbour lives in the string "123 Main Street", or Hacker News is an interesting DNS name full of great content.
That’s a very new feeling for me. I read the entire post (with no prior knowledge of BGP at all) and I got chills from thinking how deeply intertwined US companies and the US government are.
I know this has always been the case, of course, but now I have lost trust. Whatever the reasons of this "leak" were, I am not accepting any information written in this message (search for the link to another coverage of the incident in the comments).
It is quite weird and quite logical at the same time: this is the end of an era.
I remember the face of one guy after we chatted about lawful interception over a couple of drinks. He was visibly shaken like he has seen the hell through the door just opened before him.
These kinds of infrastructure is present everywhere, for a very long time. Just because not everyone is talking about the matter doesn't make it non-existent.
For example, in 2003, I saw how Japan monitored their network traffic in real time. It was eye opening for me, too. Technologies like DPI which required beefy servers are now trivial to implement with the right hardware.
can confirm this is true - a single rack of servers can now handle terabits of traffic.. in real time with near zero added latency, anti-ddos companies do this as a service.
It used to be that they needed to dedicate entire rooms for interception hardware, and tighter maintenance schedules. Nowadays, the devices they use are tiny in comparison, way easier to hide. I've encountered infrastructure companies discovering hardware that doesn't belong to them, in their local infrastructure, and when detected and reported, law enforcement came to pick it up, and refused to talk about it. That case still hasn't had a resolution, and it's about 4 years ago now.
Because it wasn't in the US, and the specifics don't really matter. All countries I've lived in so far has had similar capabilities for sure, and practiced them too.
the point is, this isn't the action of local authorities. this is state level activity. if it is local, that's a level of sophistication and corruption that I have ever been aware.
Just for context in many smaller countries outside the US there isn't that much of a "local" thing like there is in the US, I.e. the national authorities may handle a lot of stuff that may be done by the local authorities in the US
That's OK and fair I think, even as a European. HN is fairly US-centric, both submissions, users and comments. I think after more than a decade here, you get to used to everyone assuming you're American and capitalist by default, which given the company who owns HN, kind of makes sense ;)
They may not be able to decrypt it now, but it is well known that most of encrypted Internet traffic is permanently stored in NSA data centers [1] with hopes of decrypting it soon once quantum computing can do it.
I thought the whole point of the acme client was that the private key never leaves my server to go to let's encrypt servers. Now yes, if I am using cloudflare tunnel, I understand the tls terminates at cloudflare and they can share with anyone but still it has to be a targeted operation, right? It isn't like cloudflare would simply share all the keys to the kingdom?
no, the private keys are yours - the root CA just 'signs' your key in a wrapper that is was "issued" by ex: letsencrypt, and letsencrypt just has one job: validate that you own the domain via acme validation.
they would just compromise wherever your tls is terminated (if not E2E which most of the time it is not), but also just taking a memory dump of your vm / hardware to grab the tls keys and being able to decrypt most future traffic and past is also an option.
It's funny that people still have any expectation of privacy when using a vm hosted at a place like AWS or Azure... They're giving any and every last bit you have, if the right people ask.
It isn't just aws though. You could say exactly the same about digital ocean or linode.
Even if you have your own rack at a colocation, you could argue that if you don't have full disk encryption someone could simply copy your disk.
I am just trying to be practical. If someone is intent on reading what users specifically send me, they can probably find bad hygiene on my part and get it but my concern is they should not be able to do this wholesale at scale for everyone.
> if you don't have full disk encryption someone could simply copy your disk.
You can have full-disk encryption then. It can still possibly be compromised using more advanced methods like cold boot attacks but they are relatively involved, and is very detectable in the form of causing downtime.
actually, even the CTO of AWS couldn't hijack an abusive VM server because legal did not allow them to, but when the government is asking it I guess that all flies out of the window.
Pretty much as you say. Legal exists within a system of laws. Hypothetically these laws might not have a carve-out for "CTO doesn't like the behavior" but they almost certainly do have a carve-out for "national security reasons". You'll pretty much never find a lawyer advising a client to break the law because it would be more ethical to do so.
who knows how often or what kind of access is/can be given, but we will never know most likely because National Security Letters are almost always accompanied with gag orders
It's crazy that it seems like we're just going in loops every decade or so. New people enter tech, mostly focus on their own stuff, after a while, it becomes very clear how "deeply intertwined US companies and the US government are", and these people now lose their trust. Eventually, things been going well for some years, so new people enter the industry, with the same naive outlook, thinking "This couldn't be true of the government we have today" yet eventually, even they realize what's going on. Rinse and repeat every last 3 decades, and that's just what I remember, I'm sure others remember even further.
As someone in networking, it checks out, and I also know the author.
Imagine an overworked, underpaid, network engineer. Mistakes happen. This time though, the entire world is hyper fixated on what amounts to an easy to make mistake and now your mistake is in the intel briefs of 50 countries. Oops. Rough day at the office.
The magic of the system is that the ratio of new entrants who don't aren't yet jaded enough to not be useful idiots vs the rate at which people become jaded vs the rate at which those jaded people leave makes it self sustaining.
This is... hard to follow. You seem to be implying that Cloudflare is covering for USG's failed military op-sec surrounding a malicious BGP leak, and judging that this is such a bad action (on the part of Cloudflare) to undermine your trust, not only in Cloudflare, but in all companies and the US government entirely. I don't think the situation is so dire.
Cloudflare's post boils down to Hanlon's razor: a plausible benign interpretation of the facts is available, so we should give some scrutiny to accusations of malice.
Are there specific relevant facts being omitted in the article, or other factors that diminish Cloudflare's credibility? They're clearly a qualified expert in this space.
Let's assume for the sake of argument that the BGP leaks (all of them from the month of December, in fact) were the result of secret US military intelligence operations. The fact that militaries generally use cyber vulnerabilities to achieve their objectives is not news, and the US military is no exception. Keeping specific exploits secret preserves a valuable advantage over competitor states.
One could argue that Cloudflare's post helps to preserve USG's secrecy. We can't know publicly whether USG solicited the article. But even if we assume so (again assuming malice): Is Cloudflare wrong to oblige? I don't think so, but reasonable people could disagree.
Merely pointing out Hanlon's razor doesn't fundamentally change the facts of the situation. In Cloudflare's expert opinion, the facts don't necessarily implicate USG in the BGP leaks without an assumption of malice. Assuming Cloudflare is malicious without justification is just deeper belief in the conspiracy that they're arguing against.
If Cloudflare is distorting the facts, we should believe (rightly) that they're malicious. But I don't see any evidence of it.
Respectfully your comment sounds like paranoid thinking.
The section of the article pointing out the AS prepending makes it really clear the route leak is a nothing Burger.
It's incredibly unlikely this leak change how any traffic was flowing, and is more indicative of a network operator with an understaffed/underskilled team. Furry evidence is that a similar leak has been appearing on and off for several weeks.
That's not to say the US government can't, doesn't or didn't use the Internet to spy, it's just that this isn't evidence of it.
Relevant section below:
> Many of the leaked routes were also heavily prepended with AS8048, meaning it would have been potentially less attractive for routing when received by other networks. Prepending is the padding of an AS more than one time in an outbound advertisement by a customer or peer, to attempt to switch traffic away from a particular circuit to another. For example, many of the paths during the leak by AS8048 looked like this: “52320,8048,8048,8048,8048,8048,8048,8048,8048,8048,23520,1299,269832,21980”.
> You can see that AS8048 has sent their AS multiple times in an advertisement to AS52320, because by means of BGP loop prevention the path would never actually travel in and out of AS8048 multiple times in a row. A non-prepended path would look like this: “52320,8048,23520,1299,269832,21980”.
> If AS8048 was intentionally trying to become a man-in-the-middle (MITM) for traffic, why would they make the BGP advertisement less attractive instead of more attractive? Also, why leak prefixes to try and MITM traffic when you’re already a provider for the downstream AS anyway? That wouldn’t make much sense.
People are so wanting to believe there was an advanced cyber attack to Venezuela’s grid and ISPs that they forget this is a country that hasn’t updated its infrastructure in more than two decades while also not providing any significant maintenance. Most of the “new” technology deployed at the state and federal level comes from corrupt foreign and domestic “suitcase” companies that charged a lot of money to deliver poorly designed systems often even lacking the as-sold equipment. So Venezuela isn’t precisely the most formidable adversary when it comes to cybersecurity.
People also wanting to believe there was even a need for sophisticated cybersecurity attack in the first place. In a country where average household income is around $230 per month. In much wealthier country like Russia you can literally buy dump of all possible leaked data on any person for $1 and for $100 you can get all information government have about a person including camera and mobile phone tracking, etc.
And Venezuela is very very corrupt country. No cyberattack needed when you can pay $10,000 - $100,000 for a dude to pull the lever or to forget to pull the lever and literally 99.99% of people in a country do it.
Though these theories are easy to explain because people in mostly US community like HN have no understanding of what total corruption look like in a shit hole countries.
This is CANTV they are talking about. This is the company I requested a new phone line from and it took 9.5 years to get it installed.
After waiting for 3 years, I gave up ended up paying one of their technicians I randomly found working in the street. He gave me a phone line that apparently used to belong to a taxi company, judging by all the wrong number calls I got. All that just to get 4mbps DSL service in 2019.
Last year, out of nowhere, I finally got a call from the company saying they were ready to install it.
Thankfully, a bunch of companies appeared out of nowhere (a lot of them with links to people in the govt, surprise) in 2020 and we got fiber.
Oh and a couple of years ago, my parents "lost" their phone line and have been without POTS ever since. Maybe it's karma for me paying for a phone line all those years ago...
1. There was a cyber attack on the Venezuelan power grid. This disrupted comms coming into the attack and made it much harder to coordinate a response.
2. It was not in any way related to this BGP, of which, as someone in networking, looks like a simple and fairly common mistake. It wouldn't really buy them anything anyway, the breach happened 6+ months before.
I once was half way through a road trip when google maps routed me off the highway, through a walmart parking lot, and onto another highway.
I assumed it was a badly performing algorithm. But if it had instead routed me through a McDonalds drive through, I'd have assumed it was foul play.
I think the article makes a decent case that this was the former and not the latter, though it would be interesting to see route leaks visualized on a map over time. Too many odd coincidences could sway me the other way.
Truth. There's been a lot of work over the past 10-15 years to strap on best practices and validation to make these kinds of incidents less common or impossible. The article even talks about several upcoming changes/standards at the end.
Scary that so much of the basic internet infrastructure is being managed by US companies. Maybe now the rest of the world will change and become more independent. We should have learnt our lesson long ago though.
Considering that the internet was invented and built from scratch by the US military, US universities, and US companies, why are you surprised? And who do you suggest could or should manage much of the internet backbone, if not them?
The rest of the world exports its talent to the US because they don't pay enough. There's no reason why the EU couldn't have made an Akamai or Cloudflare clone decades ago save for the money.
Whether the claims are true or not, this was a very entertaining BGP refresher. It made me wonder: 15+ years ago, I was network engineer and we used quite a bit of "BGP community magic" to get the routing outcomes we wanted.
If BGP only really needed to represent three types of peers (provider, customer, actual peer), wouldn't BGP configuration and perhaps even BGP be massively simplified?
By analogy: i could massively simplify google maps direction algorithm by getting rid of all that annoying and unnecessary traffic information, annoyingly complex labels about speed limits and lane count, and all the data points about stop signs, traffic lights, and so on. Its just a path-finding algorithm after all and all that extra info just makes for more computation and complexity. Who cares if it mean all the traffic for a major metro goes across a 1-lane bridge and leaves all the other roads empty.... its the shortest path, what could go wrong?
I have been looking into BGP incidents for a while, and one of the things that continues to puzzle me is figuring out the difference between legitimate outages and noisy but expected behavior. ~
The mental model I’ve been using is: Intentional change (maintenance, policy update) Accidental leak (misconfig, partial rollout) Structural failure (dependency or upstream issue) I like to ask three questions first: Did the blast radius grow over time, or did it appear instantly? Did paths change symmetrically or only in one direction? Did things revert cleanly or drift back slowly? Some concrete tricks that helped: Look for AS-path prepending changes first. Compare visibility across regions rather than just globally.
Track “who benefits” from the new paths, even if only for a short time. I’m interested in how others approach this: What is your first indicator that things are indeed wrong? Do you prefer automated alerts or manual recognition of a pattern?
That the relations between EU and USA are not in an historical maximum, no discussion... but enemy? The hyperbole is a little big too far fetched.
> I hope EU companies will stop manufacturing US airplanes and other things.
Independent of how little we may like current US politics: a) it will probably change, more sooner than later. And b) starting a trade war with the US is not very good idea. We like it or not, there are many things that we need desperately to be able to produce. Starting with computers and SW. And please don't start with "OOS SW" as much as I like the idea, and I constantly advocate for it, even if we start yesterday, it will take decades to build everything again.
Though I think the EU is thinking that blowing up critical energy pipelines and seriously damaging Europes economy through the resulting much higher energy prices wasn't too friendly.
All to boost US energy exports, make US manufacturing more competitive and met US geopolitical goals.
ie that's not declaring war - but's it's a pretty big FU wake up call. Turning a blind eye when the US treats central and south american countries with contempt is one thing, but it's a bit of a shock when it openly does the same to you - cf Greenland as another example.
> Though I think the EU is thinking that blowing up critical energy pipelines and seriously damaging Europes economy through the resulting much higher energy prices wasn't too friendly.
Although if you ask "cui bono?" there are some pointers in that direction, is not proven, and there are also pointers in other directions. I refrain of accusing without reasonable proof.
>All to boost US energy exports, make US manufacturing more competitive and met US geopolitical goals.
I cannot blame them for that. Of course anything they do is in their benefit. Some may argue, is precisely what the government should do. It is clear, that while the action in Venezuela was against a very shady government, was done thinking in US interests (as can be clearly seen by titles as "Trump Says Venezuela Will Buy Only US-Made Products From Oil Deal Proceeds").
So yes, they do all in own interest, and the EU isn't and wasn't very different. Alone if you consider the long colonialism years. Now the EU is acting poorly, but I would say not out of altruism, but incompetence and bureaucratic stagnation.
I will not say "is impossible" they do the same in Greenland... For good reason I think. BUT comparing the 2 is also farfetched. I know plenty of people from Venezuela, and unless you were part of the government, you were strongly against it. I know no venezuelan (from the at least 100 I know) that wanted Maduro there. And many are still in party modus. Granted, I know primarily expats, so, survivor bias may apply... but still.
Ruling by oppression does not make you friends until eventually you may find yourself alone. The US government and hyperscalers may think that in the coming decades they can take on the rest of the world on their own, and I'm not looking forward to that future materializing.
> Granted, I know primarily expats, so, survivor bias may apply... but still.
Exactly - the views of people who have left Cuba, Venezuelan, or Iran are typically not representative - by definition they chose or were forced to leave.
Indeed if they have left - why are their views informing armed intervention - should Italian American's force political change through American might in Italy over the people that still live in Italy?
It's all just performative - bottom line Trump doesn't care about good governance and democratic in Venezuela - indeed he has just come out against fresh elections - all he cares about is the flow of money and resources.
But this isn't something unique to Trump - just look at the history if US meddling in central and south america. Democracy and the will of the people ( whatever that is ) isn't the driving factor.
BTW totally accept Europe has a very similar past, and to some extent present - and you could argue that the fact that the EU is less involved in this sort of thing these days is a question of capacity rather than desire.
However that's rather my point - in a globalised world - the differences in power will equalise meaning whether countries like it or not just going around doing what you want is going to no longer be an option - and it's better to gracefully accept that and adjust rather than rage against the dying of the light and inviting in the four horsemen.
Again, that is not speaking well of the acting government. Is just not normal that so much people choose or even worst are forced to leave. That just does not speak well of that regime. Does it? So dismissing their opinions does not seem to be a useful reasoning. I know from friends of mine in many different places in Latin America (mind you, in both "left" (brasil/MX) and "right" (Arg/ Chile) countries) that there are literally thousands of venezuelans in exile. That is not normal and is not a good sign.
> bottom line Trump doesn't care about good governance and democratic in Venezuela
Totally agree. But as I said, is not "Trump". Is not a person, is institutional. Which you could reasonably argue is much worse. But OTOH, there are many people again, does not matter if they went or stay... many people from that very country that are very happy with the intervention...
My way of seeing it is: we have to wait to be able to weight the prons and cons. WWII + Plan marshal was basically the same, wasn't it? And I'm pretty happy with the results and how everything played out...
> Again, that is not speaking well of the acting government.
Being forced to leave in itself is not a bad sign - do people not flee the US to Mexico to avoid justice? Perhaps they were part of an old corrupt cabal running the country for their own benefit?
Or perhaps they had to leave because of dire economic circumstances largely caused by foreign sanctions rather than internal mismanagement?
Let's be clear I'm not a fan of the current governments in the countries I've listed, but then I'm not a fan of Trump either. In neither case does it justify military invention - I'm not advocating abducting Trump to free the America people from a leader who sends troops on to the streets in cities of his political opponents.... and openly ignores the constitution.
> But OTOH, there are many people again, does not matter if they went or stay... many people from that very country that are very happy with the intervention...
The whole point of democracy is you don't have people like you and me making arbitrary choices from afar based on hearsay - and if there isn't democracy - in my view it's still a democratic choice to decide whether the cost of a rebellion is worth the price. Outside countries shouldn't be making that choice for other people ( We decided that your son dying is a price worth paying for a change in political system ).
Note that doesn't mean you shouldn't stand for your values and be assertive - and driving very hard to ensure no military inbalances. However that's a long way from self-interested coups under pre-texts.
> WWII + Plan marshal was basically the same, wasn't it? And I'm pretty happy with the results and how everything played out...
A closer comparison would be the 1953 coup to remove a democratically elected government of Iran in an attempt by UK/US to get back control of the oil. The installation of a non-democratic autocrat who was friendly to the west directly lead to revolution and the situation today.
It’s disingenuous to talk about Maduros popularity without mentioning the brutal sanctions placed on it by the US, which caused the crisis in the first place.
Literally all the people I know from Venezuela, even supporters of Maduro will tell you it started long before. Like 20 to 30 years ago. Even by the period of Chavez, or even before that. I think you don’t know very much about Venezuela. And is very disingenuous state opinions without enough knowledge and lots of ideology.
Every Canadian I know will chime into conversation about how we'll resist when your batshit crazy authoritarian leader decides to invade us. Your government is behaving like an enemy. It's not hyperbole when they're actively waging economic warfare and making worst threats.
USA an enemy to EU because of Venezuela? LMAO, EU has said nothing. In fact we agree. We have no relations to Venezuela. Now if the USA attacks Greenland, that is different.
It seems the be explicit US policy to force people to choose between China and US.
This seems to come from the US obsession with hegemony as the only strategy ( without realising that only works for 1 out of 200 countries ) - everything is framed as a US/China tussle for top dog.
Note this isn't a purely a Trumpian thing - he is just being more open/less subtle about it.
The US has to realise that it's days of global dominance are coming to and end - just as the UK had to ~100 years ago. What I hope is this time we won't have a couple of world wars during the transition to a multi-polar world.
The interesting bit is that Trump and many of his supporters seem (to me) to be openly working to bring an end to US dominance and promote the likes of China to the top spot instead. The US got its global power by being relatively undestroyed by WWII, and thus both willing and able to pay to rebuild the rest of the world, while performing some very sneaky currency and political manipulations. Now the US wants to cut off our allies and strengthen our enemies.
It is the US and its recent actions, including its rhetoric, that are driving people towards China.
Also, only reacting to US aggression after Greenland is attacked? Not prepare at all and then write a strongly worded letter after the fact?
If, after everything Trump has done, you still think he isn't serious about annexing Greenland then you and people like you, including the eurocrats, are truly hopeless.
Or to put it another way - is the binary US/China choice as top dog a false one?
I suspect most countries would prefer a multi-polar world where the majority is dominant ( democratic ), not one particular country ( autocratic ).
ie why do we have to choose to be under the heel of the US or under the heel of China?
The US has been playing the benevolent dictator role for the last 70 years, but when faced with losing the dictator role, the benevolent facade is dropping.
The US is mistaken to think that countries not wanting US dominance is the same as wanting Chinese dominance - they, in fact, want neither.
They’re pretty good as far as I know, my country didn’t receive any of those but I’ve visited countries in Africa that have received them and they’re getting really great infrastructure and knowledge sharing from them!
China picked up a 99 year lease on Hambantota Port for one Chinese firm in Sri Lanka after they defaulted to the tune of 51Bn.
Zambia defaulted during the pandemic.
Ethiopia defaulted in 2023.
Ghana suspended payments on most external debt to try and make the Chinese debt payments in 2022/2023.
Pakistan just keeps rolling those loans into other loans so it won't default.
Loas got so bad China now owns their power grid.
Suriname defaulted in 2020.
Kenya stopped paying govt workers for awhile to make their loan payments.
"Recent reports from the Lowy Institute and the World Bank indicate that 75 of the world's poorest countries face a record high of approximately $35 billion in debt repayments to China in 2025 alone."
2026 will see more countries default to be pressed into extreme measures to make the payments.
Ok, and? What does that have to do with the USA invading deposing couping 90% of countries in south america, starting wars in the middle east, and saddling all of africa with debt via the IMF and WB?
I guess it comes down to who you your government wants to be in debt to, and if you're willing to potentially starve to death so your government can pay off the loan.
Just when you think it's only the evil dictatorial regime trying to break up NATO along comes a helpful, presumably USA-based HN commenter, to remind you that a lot of USA citizens are also supportive of destroying any semblance of accord with former allied nations and spreading disharmony wherever they go.
It's right for all of us to consider whether our online presence supports fascist dictators be they from USA, Russia, Venezuela, or wherever.
Thanks for the reminder that even here people are sometimes shit, I guess.
I'm pretty sure you know what the parent poster meant, and you should take it as a compliment to you and our HN community that they didn't intend you, or us, to be included in that definition of 'anything US-based'.
Stirring the pot like that isn't helpful to you, to HN itself, or any of our community.
Hackers -the ones from MIT- and smart people hate the US goverment snooping and Echelon like projects more than Europeans. These were either privacy first since forever. If not, they fought hard to say whatever they liked without consequences.
In current times, encrypted accesses to Usenet and IRC via I2P and the like will boost the platforms more than ever. Why? IRC and Usenet are dead simple, Emacs has ports to everything and among being a Lisp env, an editor and a minimal web browser, it's an IRC and Usenet client too even under Android. Oh, and you can set I2P under Android too. Thus, you just have to set Emacs against it. There are several guides online.
Rocksolid BBS' federate with the whole Usenet (and some newsgroups catch anything text based, no binaries). On I2P proxies to Libera.chat, it's just a matter of time to exist. Meanwhile, there's ILITA IRC with I2PD.
Difficult? A Chinese Bluetooth keyboard it's worth very little today, and the gains are enormous. You can chat with people with really small bandwidths but encrypted either with TLS or I2P. You don't have blocks (except bans under IRC), comment limits, enforced timelines and any enshittification coming from social networks. Also, you can short Usenet threads by score. That's it, it's there a brilliant poster and comp.misc, you set the score for all his comment to 1000; then that random Joe will always be on top in any group. Try that with X/Twitter, Reddit or whatever.
You will be able to chat with Western Europeans, Eastern Europeans, the Japanese, anyone. Forget tribes, forget the bullshit made to earn zillions of cash from X with shitty fabricated polemics. Forget Meta's snooping and industrial stealing. You aren't enforced to give your real name and address.
It is perfectly normal for an anycast network operator to have multiple sites from which they make BGP announcements (which is how anycast works in the first place), which gives them multiple vantage points for this sort of analysis.
Other CDN companies can do it too, it's just that they don't work on signalling their engineering focused organization.
This was a while ago but I think it was Akamai that pioneered that. I remember how impressive it was in the early oughts though we take it for granted now.
Until their systems block you for no reason. I recently had a similar issue on a work related site. Fortunately, I was able to reach to the administrator (which is on another country) and had the knowledge to write a report which was useful enough for the said administrator.
And this is for a system which has the same static IP which is not shared with anything for 10ish years.
I recently, with great reluctance, had to put a personal site behind Cloudflare free option. It gets lots of use, but brings no revenue (costs me to run) and I have little spare time.
Found out that I was blocked from it in my default setup. Firefox with default settings, and no VPN.
I'm working hard to turn Cloudflare off.
Cloudflare is not remotely awsome. It's also a solution to a problem (aggressive scrapers that produce DOS) which is worse.
Just very high usage all of a sudden, after years of reasonable usage. Google has indexed it (respectfully) since 2008 just fine.
New traffic isn't humans. I blocked some AI scraper user-agents, which helped, a bit. But most new user agents are identifying as vanilla browsers, not scrapers.
I don't have numbers. It was enough to consume all nginx worker_connections. Raising the number doesn't help, as it's just reverse proxying to JVM.
After the switch, Cloudflare showed USA and Singapore as heavy traffic sources.
I don't mind scrapers on the site, but app is a search engine (of sorts) so every page view consumes some CPU. Including 'facet this search' buttons. My (WIP) solution is to rewrite to make it all client-side and put it all on a CDN.
The post mentions a number of times that leaks happen "all the time", but the only comparative data shown related to this is for historical leaks from AS8048.
Does anyone have data on what the general frequency of these leaks is likely to be across the network?
I’ve seen leaks impact my company directly 4 or 5 times in 4 years, so I would think often enough since we own a /9~ and don’t change our routes too often.
BGP is outside of my skillset, and I'm sure the analysis is fair and accurate. However, had billion dollar US based company Cloudflare detected widespread manipulation of routing tables by the US secret services, I certainly wouldn't trust them to publish it.
This is a good opportunity to assess what parts of your own online activity could be impacted by an attacker in the middle (assisted by a BGP leak or otherwise) and, if you're a service provider, how you can protect your customers.
At first pass you probably use HTTPS/TLS for the web, and you know that you shouldn't click through invalid certificate warnings. So the web, tentatively, looks pretty safe.
Email jumps out as vulnerable to eavesdropping, as we largely use opportunistic encryption when transferring messages between mail servers and an on-network-path attacker can use STARTTLS stripping or similar techniques. Most mail servers happily send using cleartext or without validating the TLS certificate. Check that you and your counter-parties are using DNSSEC+DANE, or MTA-STS to ensure that authenticated encryption is always used. Adoption is still quite low, but it's a great time to get started. Watch out for transactional email, like password reset messages, which virtually never validate encryption in transit (https://alexsci.com/blog/is-email-confidential-in-transit-ye... ; instead use multi-factor encryption).
TLS certificates themselves are at risk, unfortunately. An attacker who controls the network in-and-out of your DNS servers can issue domain-verified certificates for your domain; even removing protections like CAA records. DNSSEC is the classic solution here, although using a geographically distributed DNS provider should also work (see multi-perspective validation). Certificate transparency log monitoring should detect any attacker-issued certificates (a review of certificates issued for .ve domains would be interesting).
Ideally, we should build an internet where we don't need to trust the network layer. A BGP route leak would be a performance/availability concern only. We're not there yet, but now is a great time to take the next step in that direction.
Slightly off topic, but if I want to understand the concepts discussed in this article, what all topics should I learn? Is this a good starting place or enough to understand everything in this article - https://beej.us/guide/bgnet/
Because of the formulation of the question, I assume (please don't be irritated, if wrong) that you have very little knowledge of networking. In that case, it won't harm the Beej guide, but probably not the best point to start. The article discuses BGP protocol, which is totally absent in the linked guide. You may write literally millions of networking applications without need to know anything about BGP. Only if you are working in the backbone of the internet, you will encounter BGP, not even in big private networks. It will be a long way to really start from 0 up to BGP.
This is what helped me back in the day when I did that stuff. I didn't know my ass from a hole in the ground and Avi's guides got me up and running, fully multi-homed!
First learn the basics of IP networking, looking for CCNA certification materials or Comptia Network+ materials would give you a good base of knowledge.
BGP is just the way networks exchange routes (paths to a specific IP address) between each other. Once you know a little bit about how routing works it will make sense.
BGP itself is fairly simple to understand and entirely disconnected from day to day networking, you could start on it with very minimal understanding of anything else.
It's the protocol used by carriers to route traffic globally. (Make automated decisions about which core router should receive the traffic coming out of your AS ("network", kinda) )
It's entirely detached from anything else so you're pretty unlikely to have heard of it. In that way it's similar to SS7.
The Internet is a network of Networks, BGP (Boarder Gateway Protocol) is how routers tell other routers what networks they are connected to. This allows you to connect to any device on the Internet, even if you have to go through 5 different networks to get there.
> As news unfolds surrounding the U.S. capture and arrest of Venezuelan leader Nicolás Maduro, ... It is also noteworthy that these leak events begin over twelve hours prior to the U.S. military strikes in Venezuela. ...
This is how I imagine Russian companies in Russia write about the Russian war on Ukraine.
The comments here surprise me a bit. The common thread so far seems to be a general fear of US based companies, but how is that relates to the article?
Cloudflare's post is pretty boring here in that regard. They dig into how BGP works and propose that similar leaks seem common for the Venezuelan ISP in question.
Sure they could be wrong or even actively hiding the truth of what happened here, but the article mentions nothing of Cloudflare being involved in the action and they're describing a networking standard by pointing to publicly available BGP log data.
What am I missing here that everyone else seemed to zero in on?
I don't think this article provides any evidence of anything to be scared of.
That said, based on what we know already, there is no reason to take everything is this article at face value necessarily.
Firstly, if anybody isn't aware of the history of Stuxnet, it's worth reading, because otherwise you'd underestimate the government's ability to use 0-days by an order of magnitude (we're talking full custom-written multi-month hacking projects with root-kits and custom fake drivers delivered successfully to an airgapped system, source wikipedia). Also worth learning about Dual EC DRBG debacle.
Secondly am immediate friend of mine worked at a FANG company that routinely sent a firehose of all sorts of things matching all sorts of filters directly to governments. In fact many ISPS have back-doors built in and that's not really disputed (wikipedia: room641A).
So the question to ask yourself is -- if this was a deliberate interaction that cloudfare was required to participate in via a warrant, would they legally even be allowed to publish a blog post that contradicted this?
So I think that is probably the default attitude of skepticism you are seeing, which in my opinion is a good default. Plus the primary claim of this article "Look it wasn't 1 routing issue, it's been happening for even longer! Therefore nothing to look at here!" seems really weak.
> So the question to ask yourself is -- if this was a deliberate interaction that cloudfare was required to participate in via a warrant, would they legally even be allowed to publish a blog post that contradicted this?
So you're proposing they could be in a situation where they can either:
1. Publish an untruthful blog post, relying on public data available from multiple parties, trying to somehow explain it all while avoiding talking about their involvement in a way that would get them in PR, legal or political hot water; or
2. Publish nothing.
And they chose #1?
The only way #1 makes any sense at all is if some greater consequence to not publishing was put in place. But that would be more something like "the US gov essentially forced Cloudflare to write this" than "Cloudflare was part of this".
Unless they were part of this, _and_ the government forced them to write a post saying they're _not_ part of it and...
For my money: this is something in the news making it a good marketing opportunity which is ultimately what the blog is--trying to market Cloudflare and the brand to technical crowds.
For me number 1 is difficult basically because of who runs Cloudflare. I trust Matthew Prince because I find him to be: consistent and credible.
I work in go to market, specifically for businesses like Cloudflare, I can and have said "this real world situation is going to have resonance for the next 5-10 days, what is the lowest cost blog post you could publish that is related?" - because I only manage teams who produce content that is genuinely, at some level, value add or interesting to my target market, you would end up with a blog post exactly like this. In fact, this blog post is doing that job, here we are, cloudflare users, discussing cloudflare.
Does it work out if we imagine that Prince and/or the person who wrote the post don't have the full picture of Cloudflares own involvement?
It becomes nuanced doesn't it? First thing is: to trust him fully is to understand what it means to trust him... that he knows his business well enough that he can intuitively feel things are wrong. That comes from not being checked out, so: he knows who is in his company and why, he knows the types of projects happening in his business and why, he has easy levers to gain real time information when something feels wrong, and - he monitors his business correctly. I trust Matthew because I know him, so I believe all those things are true. The final part is that trust is also about knowing that mistakes happen, and that they are being: sought out, addressed and owned. So when I say I trust him, it's because I believe everything aforementioned - it makes your scenario safe, at least to me.
> if some greater consequence to not publishing was put in place.
Such as, losing trust,
due to this being the one postmortem you don’t write about?
> "Look it wasn't 1 routing issue, it's been happening for even longer! Therefore nothing to look at here!" seems really weak.
It's actually really strong since it implies that there's no real time-based correlation with the recent action in Caracas. Especially as the purported correlation was rather weak to begin with.
It's even older than Stuxnet, but either Dish Network (Echostar) or DirectTV did something similar in the early 2000's/late 90's.
They were having a lot of trouble with pirate receivers, so they added small chunks of code to normal device updates and this went on over a period of weeks/months. On the final update, it stitched all those bits of code together and every receiver that wasn't a legitimate one displayed the message "GAME OVER" on the screen and stopped working.
Obvs it was a long time ago so forgive me if I get some details wrong.
I looked at this a couple days ago and my thoughts were basically the same as Cloudflare's. It looks like a misconfiguration - one that's easy to make and isn't terribly uncommon. I can't rule out it wasn't an attack, but absent some other evidence, I don't see any reason to believe it was one.
That said, looking at their Cloudflare radar page now for AS8048, I don't recall there being any other BGP route leaks listed there for December from AS8048 and I definitely don't recall there being any BGP origin hijacks listed. The latter is something rather different from a route leak - that looks like someone blackholing some of CANTV's IPs.
I don't think I somehow just missed that since I definitely looked at CANTV's historical behavior to see if anything they did was unusual and that would have been one of the first things I checked, but perhaps they updated radar with data from other collectors or re-ran anomaly detection on historical data.
Ah yes, and we're back into "but my buddy told me " if you have to say that then your story just isn't worth saying or hearing and you should reconsider how impervious you are to conspiratorial thinking
The one thing they relied on "my buddy told me" for is actually not really in dispute as they say. Between CALEA, the Snowden leaks, and the earlier stuff (like the beamsplitters in Room 641A), we have known clearly based on a number of public and verifiable sources that the US government has its fingers deeply into the data streams that flow through US companies. This is a reasonable inference even absent all of this information.
Now ... I don't think any of this actually supports the parent comment's implication that Cloudflare took some anti-Venezuela action at the request of the US government, just that your criticism is kinda unfounded.
I share your view - how does this article imply US companies and/or government involvement? If there were such involvement what aspect of BGP gives the US entities more ability to carry this out vs other nefarious actors? I ask this sincerely knowing almost nothing about BGP and wanting to learn...
You may have missed https://news.ycombinator.com/item?id=46504963 a few days ago where this same anomaly was discussed and American government involvement was directly implied by the article.
The top comment of that thread points out exactly the same thing this Cloudflare article does; that there doesn't really seem to be be any indication this was anything nefarious.
Probably because most people only read headlines (and maybe 3 paragraphs) combined with the fact that the US has a long history of doing what people are condemning them for, even if this particular instance probably wasn't a case of such behavior. Especially considering how the general sentiment towards the US has gotten bitter with constant threads of invasion of Denmark and Canada by their government.
Or it's just Russian and China socket accounts? Who knows...
There was another post a few days ago that suggested a connection between the American invasion of Venezuela and the BGP anomaly: https://loworbitsecurity.com/radar/radar16/
Combine that with the news of Trump publicly admitting that the US is willing to take military action to bring other countries in line, even against their own allies: https://edition.cnn.com/2026/01/06/politics/us-options-green...
Personally, I don't think the Americans would bother hide their attack and make it look like an accident under the current regime. Trump would announce the CIA/NSA/FBI/whatever did the Greatest Attack, and Amazing Attack, to Completely Control and break the Weak Government of Venezuela to Rescue Their Oil. I'll believe the "it was just a misconfiguration" explanation for now.
I think it only makes sense that people start fearing the influence of American companies given the current developments. When America is in the news, it's either threatening someone, pulling out of cooperative efforts, or delivering on a previous threat. That's bound to derail discussions whenever American companies are involved and it'll only get worse with the way things are developing.
That's what I find interesting about the billionaire elite standing behind el presidente, like, sooner or later he'll be gone and you guys -and your companies- won't. There's been no more compelling argument to actually overtax the rich to give to the masses than the last 13 months.
Eat the rich. History won't forget.
I think it’s just bog standard, “USA bad, not USA good” thinking.
I mean, it's the context around the article...based on recent events...
I'm half sleepy but I liked the post. The analysis regarding path prepending really drives the accident theory home. If a state actor were trying to intercept traffic (MITM), the last thing they would do is pad the AS path multiple times because that tells the global routing table, "Don't come this way, I am the long scenic route" lol
This could be a classic fat finger config error, most likely a route map intended to manipulate traffic engineering for their own upstream links that inadvertently leaked widely because of a missing deny-all clause. Neverthless, a good reminder that BGP is still fundamentally a trust based system where a single typo in a config file can cascade globally. Never attribute to malice that which is adequately explained by a missing export filter.
> If a state actor were trying to intercept traffic (MITM), the last thing they would do is pad the AS path
That's presumptuous: A state actor would (and could trivially) pad the wrong directions to flow traffic down to pops that are not making new announcements (and thus not-implicated by cloudflare and other "journalistic" efforts).
There's also a lot between fat-fingers and deep-state: I know of some non-state actors who do this sort of thing just to fuck with ad impressions. I also doubt much usable intelligence can be gained from mere route-manipulation thing, but I do know that if it is a fat-finger, every techdude in the area was busy at that time trying to figure it out, and wasn't doing their best work twelve hours later...
> most likely a route map intended to manipulate traffic engineering for their own upstream links
...that being said, this does seem plausible: Most smaller multihomed sites I've seen (and a few big ones!) have some kind of adhoc health monitoring/rebalance function that snmp or something and does autoexpect/curl or something-else to the router to run some (probably broken) script, because even if your uplinks are symmetrical, the rest of the Internet isn't, so route-stuffing remains the best way to manipulate ingress traffic.
> Never attribute to malice that which is adequately explained by a missing export filter.
As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them. Once I have a third, I have a legitimate need to manipulate my own ingress.
The problems with the BGP are legion, and not just one thing that prevents BGP and security from sharing time in a sentence.
> A state actor would (and could trivially) pad the wrong directions
This isn't how BGP works. An AS-PATH isn't the path the traffic will follow; it's the path that this overall announcement has allegedly tranversed and is (one of many attributes) used to judge the quality of route. The next hop tells our peer where they should send the data if they like this route.
Putting more things in the AS path makes the route less attractive. Leaking a new route isn't going to magically make some other route become more preferred.
You're spot on regarding the mechanics. It's important to reinforce that in BGP, AS-PATH length is a cost metric and not a steering wheel.
Actually many networks will prefer routing over a cheap AS path no matter how long it is.
> > and is (one of many attributes) used to judge the quality of route
Lower cost usually means lower quality and is an example of how a long path being leaked can result in traffic flowing away from high quality path to the leaked path.
Not saying that this is the case with Venezuela, just explaining the reality of BGP where path prepends are often ignored.
His claim-- as best as I can read it-- is that B leaking a long-length route changes where traffic is routed, but not to B.
It's possible he's saying something else, but I can't figure out, and he hasn't clarified.
> This isn't how BGP works
This is exactly how BGP works.
https://bgplabs.net/policy/7-prepend/
> Leaking a new route isn't going to magically make some other route become more preferred.
Not magic, but technology can look like magic when you don't understand it.
> > > That's presumptuous: A state actor would (and could trivially) pad the wrong directions to flow traffic down to pops that are not making new announcements
> > Leaking a new route isn't going to magically make some other route become more preferred.
> Not magic, but technology can look like magic when you don't understand it.
Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
I've used BGP for over 25 years, so I'm really curious what you're thinking. Or if you're describing something else, you're being really unclear.
Or if you're just describing withdrawing a route and replacing it with a really undesirable route -- sure, we do that all the time. But that doesn't match this scenario and isn't going to get flagged as a routing anomaly.
> https://bgplabs.net/policy/7-prepend/
You know what's really toxic? Not explaining what you mean and just sending some introductory lab documentation about what the other person has already clearly shown they understand.
I don't even know what you mean by a lot of these things.. e.g.
> > > As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them.
A straightforward reading of "forward" doesn't work for this sentence. I should not take a route from peer A and send it to peer B. Peering isn't transitive. If I try, it should be filtered.
Peering means to give your own routes (and your transit customers' routes) to someone else. Not your other peers routes.
> Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
> ... I'm really curious what you're thinking
That the actor actually wanted the traffic to flow over route C.
> You know what's really toxic? Not explaining what you mean and just sending some introductory lab documentation about what the other person has already clearly shown they understand.
I think perhaps you and I have different ideas of what is "clear", for example when you said something that is totally covered in introductory lab documentation, I thought it was clear that you did not understand.
> I don't even know what you mean by a lot of these things
That is clear! But confusing! How can you clearly understand but not know what I mean?
> Peering means to give your own routes (and your transit customers' routes) to someone else.
That's exactly what's happening here: Not every transit customer peers with every other transit customer.
> > Please let me know of the scenario where route A is preferred, undesirable, long-path route B is advertised/leaked, and as a result traffic flows over route C.
Yes, but how does advertising undesirable route B make traffic go over route C? This is why I think you're confused.
> That's exactly what's happening here: Not every transit customer peers with every other transit customer.
I am not understanding what you're saying at all. You said:
> > > > As soon as I peer with two big sites that don't peer directly with each-other, they both gotta let me forward announcements unfiltered across them.
This is the thing you are supposed to never do as a peer, and the thing that I have a whole bunch of filtering to prevent my peers from inadvertently doing.
Are you misusing the word "peer"? It's hard to talk about BGP and routing policy without using these words correctly.
I think I'm going to give up here.
> This is why I think you're confused.
I think you're confused.
> I am not understanding what you're saying at all.
And that is why; You seem to have a very strong opinion about something that you don't understand "at all" and frankly I cannot understand how that can work.
> This is the thing you are supposed to never do as a peer
So you say, but that's what I did when back in the early 2000s, and that's what the parties in the news were doing, and if you're not totally lying to me, you know this because it's the default in BGP, that's why you would say you need to:
> I have a whole bunch of filtering to prevent my peers from inadvertently doing.
because that's how BGP works. Duh.
> It's hard to talk about BGP without using these words correctly.
and I am flabbergasted you continue to persist at it, when I have even offered you "introductory lab documentation" to help.
Peering means "give our downstream customers' routes plus our own routes; receive the same from them".
Transit means "give our entire table, receive their routes plus their downstream customers routes".
You don't give one peer's routes to another. You filter to make sure you are not doing this. They hopefully filter (using data from RIRs) to make sure you're not doing it. If both parties screw up the filtering, you "leak routes" like we're discussing here.
This has been standard practice for peering since at least 1997. It is codified, among other places, in RFC7454.
> And that is why; You seem to have a very strong opinion about something that you don't understand "at all" and frankly I cannot understand how that can work.
Do you operate an AS? Are you a peering contact? I mean, I only do it mostly for funsies now but for quite awhile that was part of my job. :P
Also, still seeking an answer to this question:
> > > Yes, but how does advertising undesirable route B make traffic go over route C [that previously went over route A]? This is why I think you're confused.
> Do you operate an AS? Are you a peering contact?
> I mean, I only do it mostly for funsies now but for quite awhile that was part of my job. :P
I'm retired now. I wrote some about my experiences on HN a long time ago:
https://news.ycombinator.com/item?id=18535518
https://news.ycombinator.com/item?id=2727993
I set up multihoming in the US (going through ARIN assignment for ASN and PI) in the early 2000s and for another larger company in the UK (doing the same same but different) in the early 2010s.
> Also, still seeking an answer to this question:
Not sure what to tell you. I've answered this within the context of the news article, if you're asking specifically what kinds of configurations do that they're the kinds that are in that "introductory lab documentation" and if you're not overstating your credentials you should be able to understand.
It might be worth linking this document from the Snowden leaks: https://christopher-parsons.com/wp-content/uploads/2023/01/n...
"NSA Network Shaping 101". Big descriptions of ASINs, and layer 3 shaping. Written in 2007.
I don't see the relation to BGP anomalies, since this "layer 3 shaping" is basically just "if you send traffic to the IP of an AS router, it probably goes over the link of that IP". None of this would help NSA "shape" arbitrary traffic onto links they are able to tap. (I'm really not sure what exactly the point of this is, the slides talk about exfil a lot, it would seem to me like some random device sending traffic to a router is more suspicious, because normal traffic never targets routers, than hitting an actual server somewhere but idk)
In en-us education "101" is often used to refer to an introductory course in a particular topic. My inference from the fact that this _educational_ slide is called "101" is that this is a basic example of core knowledge that people in this area of work are expected to have. It therefore stands to reason that there exists a "102" or "103" course that expands upon it, as well as material going far beyond "the syllabus".
The NSA and thirteen eyes generally have detailed traffic logging capability at core internet exchanges around the world. It is reasonable to think that a good way of exfiltrating data would be by having something like an ICMP or maybe even TTL based covert channel, such that there is no chance that the sent data is ever received by the recipient. I am just speculating – but that's why I thought this was interesting.
Funny to see even the NSA makes the mistake of calling a network an ASN (maybe because it's their name backwards), which is like saying I deposited money in my IBAN, or my neighbour lives in the string "123 Main Street", or Hacker News is an interesting DNS name full of great content.
Hi, it's me, The Language Fairy
https://en.wikipedia.org/wiki/Metonymy
That’s a very new feeling for me. I read the entire post (with no prior knowledge of BGP at all) and I got chills from thinking how deeply intertwined US companies and the US government are.
I know this has always been the case, of course, but now I have lost trust. Whatever the reasons of this "leak" were, I am not accepting any information written in this message (search for the link to another coverage of the incident in the comments).
It is quite weird and quite logical at the same time: this is the end of an era.
I remember the face of one guy after we chatted about lawful interception over a couple of drinks. He was visibly shaken like he has seen the hell through the door just opened before him.
These kinds of infrastructure is present everywhere, for a very long time. Just because not everyone is talking about the matter doesn't make it non-existent.
For example, in 2003, I saw how Japan monitored their network traffic in real time. It was eye opening for me, too. Technologies like DPI which required beefy servers are now trivial to implement with the right hardware.
This is all I can say.
can confirm this is true - a single rack of servers can now handle terabits of traffic.. in real time with near zero added latency, anti-ddos companies do this as a service.
Is it the powerful servers making the difference here? Or the coveted back haul connections which have access to the data passing by?
I suppose it's both but the latter is a more scarce resource
It used to be that they needed to dedicate entire rooms for interception hardware, and tighter maintenance schedules. Nowadays, the devices they use are tiny in comparison, way easier to hide. I've encountered infrastructure companies discovering hardware that doesn't belong to them, in their local infrastructure, and when detected and reported, law enforcement came to pick it up, and refused to talk about it. That case still hasn't had a resolution, and it's about 4 years ago now.
> and when detected and reported, law enforcement came to pick it up, and refused to talk about it.
By "law enforcement", I'd assume the feds and not local. Why not just say which agency? Wouldn't this pretty much be FBI? Why use such a generic term?
Because it wasn't in the US, and the specifics don't really matter. All countries I've lived in so far has had similar capabilities for sure, and practiced them too.
okay fine. s/FBI/whateverAgency/
the point is, this isn't the action of local authorities. this is state level activity. if it is local, that's a level of sophistication and corruption that I have ever been aware.
Just for context in many smaller countries outside the US there isn't that much of a "local" thing like there is in the US, I.e. the national authorities may handle a lot of stuff that may be done by the local authorities in the US
what an incredibly USAmerican-centric comment.
That's OK and fair I think, even as a European. HN is fairly US-centric, both submissions, users and comments. I think after more than a decade here, you get to used to everyone assuming you're American and capitalist by default, which given the company who owns HN, kind of makes sense ;)
I like the adjacent conversations just as much, or more.
We(?) more or less want this to be a place of general curiosity,
perhaps revolving loosely around those things, but not tightly clung to them.
be afraid of that random raspberry pi device dangling off the switch.
just kidding, it's just backup access via the datacenter wifi.
for everyone
> That case still hasn't had a resolution, and it's about 4 years ago now.
Sure it has!
The resolution was “go fuck yourself, what the fuck are you going to do about it?”.
Y’know: respectfully.
It's the servers specifically the parallelization with more cores and better math functions like AVX512.
Let's say I have a public website with https. I allow anyone to post a message to an api endpoint. Could a server like this read the message? How?
They may not be able to decrypt it now, but it is well known that most of encrypted Internet traffic is permanently stored in NSA data centers [1] with hopes of decrypting it soon once quantum computing can do it.
[1] https://en.wikipedia.org/wiki/Utah_Data_Center
They have a relationship with your cert provider and get a copy of your cert or the root so they can decrypt the traffic.
I thought the whole point of the acme client was that the private key never leaves my server to go to let's encrypt servers. Now yes, if I am using cloudflare tunnel, I understand the tls terminates at cloudflare and they can share with anyone but still it has to be a targeted operation, right? It isn't like cloudflare would simply share all the keys to the kingdom?
Yes. They could issue their own certificates, but we have CT to mitigate that, too.
no, the private keys are yours - the root CA just 'signs' your key in a wrapper that is was "issued" by ex: letsencrypt, and letsencrypt just has one job: validate that you own the domain via acme validation.
That is not how PKI works. Your cert provider does not have a copy of your private key to give out in the first place.
Having the private key of the root cert does not allow you to decrypt traffic either.
they would just compromise wherever your tls is terminated (if not E2E which most of the time it is not), but also just taking a memory dump of your vm / hardware to grab the tls keys and being able to decrypt most future traffic and past is also an option.
It's funny that people still have any expectation of privacy when using a vm hosted at a place like AWS or Azure... They're giving any and every last bit you have, if the right people ask.
It isn't just aws though. You could say exactly the same about digital ocean or linode.
Even if you have your own rack at a colocation, you could argue that if you don't have full disk encryption someone could simply copy your disk.
I am just trying to be practical. If someone is intent on reading what users specifically send me, they can probably find bad hygiene on my part and get it but my concern is they should not be able to do this wholesale at scale for everyone.
> if you don't have full disk encryption someone could simply copy your disk.
You can have full-disk encryption then. It can still possibly be compromised using more advanced methods like cold boot attacks but they are relatively involved, and is very detectable in the form of causing downtime.
actually, even the CTO of AWS couldn't hijack an abusive VM server because legal did not allow them to, but when the government is asking it I guess that all flies out of the window.
Pretty much as you say. Legal exists within a system of laws. Hypothetically these laws might not have a carve-out for "CTO doesn't like the behavior" but they almost certainly do have a carve-out for "national security reasons". You'll pretty much never find a lawyer advising a client to break the law because it would be more ethical to do so.
who knows how often or what kind of access is/can be given, but we will never know most likely because National Security Letters are almost always accompanied with gag orders
That's why I self host.
It's crazy that it seems like we're just going in loops every decade or so. New people enter tech, mostly focus on their own stuff, after a while, it becomes very clear how "deeply intertwined US companies and the US government are", and these people now lose their trust. Eventually, things been going well for some years, so new people enter the industry, with the same naive outlook, thinking "This couldn't be true of the government we have today" yet eventually, even they realize what's going on. Rinse and repeat every last 3 decades, and that's just what I remember, I'm sure others remember even further.
I am 50 yo and did live through multiple intertwinings. This time though, it is really the end of an era. Trust has been lost.
More positively, what's your opinion on this closer look post from Cloudflare?
As someone in networking, it checks out, and I also know the author.
Imagine an overworked, underpaid, network engineer. Mistakes happen. This time though, the entire world is hyper fixated on what amounts to an easy to make mistake and now your mistake is in the intel briefs of 50 countries. Oops. Rough day at the office.
> Imagine an overworked, underpaid, network engineer.
At Cloudflare?
All the network engineers I know are overworked. Underpaid is subjective.
The Venezuelan engineer
The magic of the system is that the ratio of new entrants who don't aren't yet jaded enough to not be useful idiots vs the rate at which people become jaded vs the rate at which those jaded people leave makes it self sustaining.
If you look closely, you can see the color of the orange Cloudflare logo being slightly adjust to match a particular individual's facial color tone.
This is... hard to follow. You seem to be implying that Cloudflare is covering for USG's failed military op-sec surrounding a malicious BGP leak, and judging that this is such a bad action (on the part of Cloudflare) to undermine your trust, not only in Cloudflare, but in all companies and the US government entirely. I don't think the situation is so dire.
Cloudflare's post boils down to Hanlon's razor: a plausible benign interpretation of the facts is available, so we should give some scrutiny to accusations of malice.
Are there specific relevant facts being omitted in the article, or other factors that diminish Cloudflare's credibility? They're clearly a qualified expert in this space.
Let's assume for the sake of argument that the BGP leaks (all of them from the month of December, in fact) were the result of secret US military intelligence operations. The fact that militaries generally use cyber vulnerabilities to achieve their objectives is not news, and the US military is no exception. Keeping specific exploits secret preserves a valuable advantage over competitor states.
One could argue that Cloudflare's post helps to preserve USG's secrecy. We can't know publicly whether USG solicited the article. But even if we assume so (again assuming malice): Is Cloudflare wrong to oblige? I don't think so, but reasonable people could disagree.
Merely pointing out Hanlon's razor doesn't fundamentally change the facts of the situation. In Cloudflare's expert opinion, the facts don't necessarily implicate USG in the BGP leaks without an assumption of malice. Assuming Cloudflare is malicious without justification is just deeper belief in the conspiracy that they're arguing against.
If Cloudflare is distorting the facts, we should believe (rightly) that they're malicious. But I don't see any evidence of it.
EDIT: Clarity tweaks.
Companies in country X are often intertwined with their governments? I'm not sure this is really news.
You changed it from “deeply intertwined” to “often intertwined” to make your strawman argument
Respectfully your comment sounds like paranoid thinking.
The section of the article pointing out the AS prepending makes it really clear the route leak is a nothing Burger.
It's incredibly unlikely this leak change how any traffic was flowing, and is more indicative of a network operator with an understaffed/underskilled team. Furry evidence is that a similar leak has been appearing on and off for several weeks.
That's not to say the US government can't, doesn't or didn't use the Internet to spy, it's just that this isn't evidence of it.
Relevant section below: > Many of the leaked routes were also heavily prepended with AS8048, meaning it would have been potentially less attractive for routing when received by other networks. Prepending is the padding of an AS more than one time in an outbound advertisement by a customer or peer, to attempt to switch traffic away from a particular circuit to another. For example, many of the paths during the leak by AS8048 looked like this: “52320,8048,8048,8048,8048,8048,8048,8048,8048,8048,23520,1299,269832,21980”.
> You can see that AS8048 has sent their AS multiple times in an advertisement to AS52320, because by means of BGP loop prevention the path would never actually travel in and out of AS8048 multiple times in a row. A non-prepended path would look like this: “52320,8048,23520,1299,269832,21980”.
> If AS8048 was intentionally trying to become a man-in-the-middle (MITM) for traffic, why would they make the BGP advertisement less attractive instead of more attractive? Also, why leak prefixes to try and MITM traffic when you’re already a provider for the downstream AS anyway? That wouldn’t make much sense.
People are so wanting to believe there was an advanced cyber attack to Venezuela’s grid and ISPs that they forget this is a country that hasn’t updated its infrastructure in more than two decades while also not providing any significant maintenance. Most of the “new” technology deployed at the state and federal level comes from corrupt foreign and domestic “suitcase” companies that charged a lot of money to deliver poorly designed systems often even lacking the as-sold equipment. So Venezuela isn’t precisely the most formidable adversary when it comes to cybersecurity.
People also wanting to believe there was even a need for sophisticated cybersecurity attack in the first place. In a country where average household income is around $230 per month. In much wealthier country like Russia you can literally buy dump of all possible leaked data on any person for $1 and for $100 you can get all information government have about a person including camera and mobile phone tracking, etc.
And Venezuela is very very corrupt country. No cyberattack needed when you can pay $10,000 - $100,000 for a dude to pull the lever or to forget to pull the lever and literally 99.99% of people in a country do it.
Though these theories are easy to explain because people in mostly US community like HN have no understanding of what total corruption look like in a shit hole countries.
This is CANTV they are talking about. This is the company I requested a new phone line from and it took 9.5 years to get it installed.
After waiting for 3 years, I gave up ended up paying one of their technicians I randomly found working in the street. He gave me a phone line that apparently used to belong to a taxi company, judging by all the wrong number calls I got. All that just to get 4mbps DSL service in 2019.
Last year, out of nowhere, I finally got a call from the company saying they were ready to install it.
Thankfully, a bunch of companies appeared out of nowhere (a lot of them with links to people in the govt, surprise) in 2020 and we got fiber.
Oh and a couple of years ago, my parents "lost" their phone line and have been without POTS ever since. Maybe it's karma for me paying for a phone line all those years ago...
1. There was a cyber attack on the Venezuelan power grid. This disrupted comms coming into the attack and made it much harder to coordinate a response.
2. It was not in any way related to this BGP, of which, as someone in networking, looks like a simple and fairly common mistake. It wouldn't really buy them anything anyway, the breach happened 6+ months before.
Yeah, the US government has advanced cyberwar fare capabilities, but this BGP anomaly is not a result of, or evidence of that.
I once was half way through a road trip when google maps routed me off the highway, through a walmart parking lot, and onto another highway.
I assumed it was a badly performing algorithm. But if it had instead routed me through a McDonalds drive through, I'd have assumed it was foul play.
I think the article makes a decent case that this was the former and not the latter, though it would be interesting to see route leaks visualized on a map over time. Too many odd coincidences could sway me the other way.
The only reason BGP route leaks aren't more common is the filtering of other ISPs. It's pretty easy to make a mistake you don't intend to.
Truth. There's been a lot of work over the past 10-15 years to strap on best practices and validation to make these kinds of incidents less common or impossible. The article even talks about several upcoming changes/standards at the end.
Scary that so much of the basic internet infrastructure is being managed by US companies. Maybe now the rest of the world will change and become more independent. We should have learnt our lesson long ago though.
Considering that the internet was invented and built from scratch by the US military, US universities, and US companies, why are you surprised? And who do you suggest could or should manage much of the internet backbone, if not them?
The rest of the world exports its talent to the US because they don't pay enough. There's no reason why the EU couldn't have made an Akamai or Cloudflare clone decades ago save for the money.
What do you mean? The internet is virtually entirely decentralized. There is no one central BGP router.
This article existing at all is a bit suspicious IMO
Whether the claims are true or not, this was a very entertaining BGP refresher. It made me wonder: 15+ years ago, I was network engineer and we used quite a bit of "BGP community magic" to get the routing outcomes we wanted.
If BGP only really needed to represent three types of peers (provider, customer, actual peer), wouldn't BGP configuration and perhaps even BGP be massively simplified?
It would be massively simplified.
Simple isn't always good.
By analogy: i could massively simplify google maps direction algorithm by getting rid of all that annoying and unnecessary traffic information, annoyingly complex labels about speed limits and lane count, and all the data points about stop signs, traffic lights, and so on. Its just a path-finding algorithm after all and all that extra info just makes for more computation and complexity. Who cares if it mean all the traffic for a major metro goes across a 1-lane bridge and leaves all the other roads empty.... its the shortest path, what could go wrong?
I have been looking into BGP incidents for a while, and one of the things that continues to puzzle me is figuring out the difference between legitimate outages and noisy but expected behavior. ~
The mental model I’ve been using is: Intentional change (maintenance, policy update) Accidental leak (misconfig, partial rollout) Structural failure (dependency or upstream issue) I like to ask three questions first: Did the blast radius grow over time, or did it appear instantly? Did paths change symmetrically or only in one direction? Did things revert cleanly or drift back slowly? Some concrete tricks that helped: Look for AS-path prepending changes first. Compare visibility across regions rather than just globally.
Track “who benefits” from the new paths, even if only for a short time. I’m interested in how others approach this: What is your first indicator that things are indeed wrong? Do you prefer automated alerts or manual recognition of a pattern?
The depth and coverage that cloudflare has is crazy
Yes and that is a very bad thing for the rest of the world. Time for non-us companies especially ones not doing business in the US to migrate away.
I wouldn't touch anything US-based with 10m long pole.
At this this, US is basically enemy to EU. Good for us, we will be less dependent on US global oil police.
I hope EU companies will stop manufacturing US airplanes and other things.
That the relations between EU and USA are not in an historical maximum, no discussion... but enemy? The hyperbole is a little big too far fetched.
> I hope EU companies will stop manufacturing US airplanes and other things.
Independent of how little we may like current US politics: a) it will probably change, more sooner than later. And b) starting a trade war with the US is not very good idea. We like it or not, there are many things that we need desperately to be able to produce. Starting with computers and SW. And please don't start with "OOS SW" as much as I like the idea, and I constantly advocate for it, even if we start yesterday, it will take decades to build everything again.
> The hyperbole is a little big too far fetched.
Sure.
Though I think the EU is thinking that blowing up critical energy pipelines and seriously damaging Europes economy through the resulting much higher energy prices wasn't too friendly.
All to boost US energy exports, make US manufacturing more competitive and met US geopolitical goals.
ie that's not declaring war - but's it's a pretty big FU wake up call. Turning a blind eye when the US treats central and south american countries with contempt is one thing, but it's a bit of a shock when it openly does the same to you - cf Greenland as another example.
> Though I think the EU is thinking that blowing up critical energy pipelines and seriously damaging Europes economy through the resulting much higher energy prices wasn't too friendly.
Although if you ask "cui bono?" there are some pointers in that direction, is not proven, and there are also pointers in other directions. I refrain of accusing without reasonable proof.
>All to boost US energy exports, make US manufacturing more competitive and met US geopolitical goals.
I cannot blame them for that. Of course anything they do is in their benefit. Some may argue, is precisely what the government should do. It is clear, that while the action in Venezuela was against a very shady government, was done thinking in US interests (as can be clearly seen by titles as "Trump Says Venezuela Will Buy Only US-Made Products From Oil Deal Proceeds").
So yes, they do all in own interest, and the EU isn't and wasn't very different. Alone if you consider the long colonialism years. Now the EU is acting poorly, but I would say not out of altruism, but incompetence and bureaucratic stagnation.
I will not say "is impossible" they do the same in Greenland... For good reason I think. BUT comparing the 2 is also farfetched. I know plenty of people from Venezuela, and unless you were part of the government, you were strongly against it. I know no venezuelan (from the at least 100 I know) that wanted Maduro there. And many are still in party modus. Granted, I know primarily expats, so, survivor bias may apply... but still.
Ruling by oppression does not make you friends until eventually you may find yourself alone. The US government and hyperscalers may think that in the coming decades they can take on the rest of the world on their own, and I'm not looking forward to that future materializing.
> So yes, they do all in own interest, and the EU isn't and wasn't very different.
Of course, what connects Europe and the USA forever is that they think the same in these matters. No one can trust them.
> Granted, I know primarily expats, so, survivor bias may apply... but still.
Exactly - the views of people who have left Cuba, Venezuelan, or Iran are typically not representative - by definition they chose or were forced to leave.
Indeed if they have left - why are their views informing armed intervention - should Italian American's force political change through American might in Italy over the people that still live in Italy?
It's all just performative - bottom line Trump doesn't care about good governance and democratic in Venezuela - indeed he has just come out against fresh elections - all he cares about is the flow of money and resources.
But this isn't something unique to Trump - just look at the history if US meddling in central and south america. Democracy and the will of the people ( whatever that is ) isn't the driving factor.
BTW totally accept Europe has a very similar past, and to some extent present - and you could argue that the fact that the EU is less involved in this sort of thing these days is a question of capacity rather than desire.
However that's rather my point - in a globalised world - the differences in power will equalise meaning whether countries like it or not just going around doing what you want is going to no longer be an option - and it's better to gracefully accept that and adjust rather than rage against the dying of the light and inviting in the four horsemen.
>> they chose or were forced to leave.
Again, that is not speaking well of the acting government. Is just not normal that so much people choose or even worst are forced to leave. That just does not speak well of that regime. Does it? So dismissing their opinions does not seem to be a useful reasoning. I know from friends of mine in many different places in Latin America (mind you, in both "left" (brasil/MX) and "right" (Arg/ Chile) countries) that there are literally thousands of venezuelans in exile. That is not normal and is not a good sign.
> bottom line Trump doesn't care about good governance and democratic in Venezuela
Totally agree. But as I said, is not "Trump". Is not a person, is institutional. Which you could reasonably argue is much worse. But OTOH, there are many people again, does not matter if they went or stay... many people from that very country that are very happy with the intervention...
My way of seeing it is: we have to wait to be able to weight the prons and cons. WWII + Plan marshal was basically the same, wasn't it? And I'm pretty happy with the results and how everything played out...
> Again, that is not speaking well of the acting government.
Being forced to leave in itself is not a bad sign - do people not flee the US to Mexico to avoid justice? Perhaps they were part of an old corrupt cabal running the country for their own benefit?
Or perhaps they had to leave because of dire economic circumstances largely caused by foreign sanctions rather than internal mismanagement?
Let's be clear I'm not a fan of the current governments in the countries I've listed, but then I'm not a fan of Trump either. In neither case does it justify military invention - I'm not advocating abducting Trump to free the America people from a leader who sends troops on to the streets in cities of his political opponents.... and openly ignores the constitution.
> But OTOH, there are many people again, does not matter if they went or stay... many people from that very country that are very happy with the intervention...
The whole point of democracy is you don't have people like you and me making arbitrary choices from afar based on hearsay - and if there isn't democracy - in my view it's still a democratic choice to decide whether the cost of a rebellion is worth the price. Outside countries shouldn't be making that choice for other people ( We decided that your son dying is a price worth paying for a change in political system ).
Note that doesn't mean you shouldn't stand for your values and be assertive - and driving very hard to ensure no military inbalances. However that's a long way from self-interested coups under pre-texts.
> WWII + Plan marshal was basically the same, wasn't it? And I'm pretty happy with the results and how everything played out...
A closer comparison would be the 1953 coup to remove a democratically elected government of Iran in an attempt by UK/US to get back control of the oil. The installation of a non-democratic autocrat who was friendly to the west directly lead to revolution and the situation today.
It’s disingenuous to talk about Maduros popularity without mentioning the brutal sanctions placed on it by the US, which caused the crisis in the first place.
Venezuela was declining and authoritarian long before the US implemented sanctions.
Maduro won fair elections in 2013 observed by neutral observers. Sanctions were placed by Obama in 2014.
Literally all the people I know from Venezuela, even supporters of Maduro will tell you it started long before. Like 20 to 30 years ago. Even by the period of Chavez, or even before that. I think you don’t know very much about Venezuela. And is very disingenuous state opinions without enough knowledge and lots of ideology.
Ok, it sounds like you know biased people. Chavez was loved by the people, and Maduro inherited his legacy.
The US administration is threatening to invade the territory of an EU member state, so I'm not sure the use of 'enemy' is too hyperbolic.
> That the relations between EU and USA are not in an historical maximum, no discussion... but enemy? The hyperbole is a little big too far fetched.
You know the US is explicitly threatening with military action on Greenland, which is part of Denmark, which is in NATO, just like the US?
The US's international appeal (especially for EU countries) is crumbling by the day.
> That the relations between EU and USA are not in an historical maximum, no discussion... but enemy? The hyperbole is a little big too far fetched.
Is it, given this kind of talk from POTUS preceding the more recent threats?
> Let's be honest, the European Union was formed in order to screw the United States. That's the purpose of it, and they've done a good job of it.
And https://www.politico.eu/article/donald-trump-putin-russia-eu...
https://www.theguardian.com/us-news/2026/jan/05/a-warning-no...
It looks mutual between EU citizens and Trump.
https://www.politico.eu/article/half-europeans-see-donald-tr...
Every Canadian I know will chime into conversation about how we'll resist when your batshit crazy authoritarian leader decides to invade us. Your government is behaving like an enemy. It's not hyperbole when they're actively waging economic warfare and making worst threats.
> we will be less dependent on US global oil police
EU, Germany especially Loves Russian oil.
So congrats?
People like you drive EU to China instead.
USA an enemy to EU because of Venezuela? LMAO, EU has said nothing. In fact we agree. We have no relations to Venezuela. Now if the USA attacks Greenland, that is different.
It seems the be explicit US policy to force people to choose between China and US.
This seems to come from the US obsession with hegemony as the only strategy ( without realising that only works for 1 out of 200 countries ) - everything is framed as a US/China tussle for top dog.
Note this isn't a purely a Trumpian thing - he is just being more open/less subtle about it.
The US has to realise that it's days of global dominance are coming to and end - just as the UK had to ~100 years ago. What I hope is this time we won't have a couple of world wars during the transition to a multi-polar world.
The interesting bit is that Trump and many of his supporters seem (to me) to be openly working to bring an end to US dominance and promote the likes of China to the top spot instead. The US got its global power by being relatively undestroyed by WWII, and thus both willing and able to pay to rebuild the rest of the world, while performing some very sneaky currency and political manipulations. Now the US wants to cut off our allies and strengthen our enemies.
It is the US and its recent actions, including its rhetoric, that are driving people towards China.
Also, only reacting to US aggression after Greenland is attacked? Not prepare at all and then write a strongly worded letter after the fact?
If, after everything Trump has done, you still think he isn't serious about annexing Greenland then you and people like you, including the eurocrats, are truly hopeless.
Or to put it another way - is the binary US/China choice as top dog a false one?
I suspect most countries would prefer a multi-polar world where the majority is dominant ( democratic ), not one particular country ( autocratic ).
ie why do we have to choose to be under the heel of the US or under the heel of China?
The US has been playing the benevolent dictator role for the last 70 years, but when faced with losing the dictator role, the benevolent facade is dropping.
The US is mistaken to think that countries not wanting US dominance is the same as wanting Chinese dominance - they, in fact, want neither.
The global south would argue that there’s been nothing benevolent about pax americana.
How are those Chinese infrastructure loans going in Africa and South America?
They're leading to a bit more prosperity (albeit awfully uneven) and relative peace. Trade is a lot better than war.
They’re pretty good as far as I know, my country didn’t receive any of those but I’ve visited countries in Africa that have received them and they’re getting really great infrastructure and knowledge sharing from them!
China picked up a 99 year lease on Hambantota Port for one Chinese firm in Sri Lanka after they defaulted to the tune of 51Bn.
Zambia defaulted during the pandemic.
Ethiopia defaulted in 2023.
Ghana suspended payments on most external debt to try and make the Chinese debt payments in 2022/2023.
Pakistan just keeps rolling those loans into other loans so it won't default.
Loas got so bad China now owns their power grid.
Suriname defaulted in 2020.
Kenya stopped paying govt workers for awhile to make their loan payments.
"Recent reports from the Lowy Institute and the World Bank indicate that 75 of the world's poorest countries face a record high of approximately $35 billion in debt repayments to China in 2025 alone."
2026 will see more countries default to be pressed into extreme measures to make the payments.
Ok, and? What does that have to do with the USA invading deposing couping 90% of countries in south america, starting wars in the middle east, and saddling all of africa with debt via the IMF and WB?
I guess it comes down to who you your government wants to be in debt to, and if you're willing to potentially starve to death so your government can pay off the loan.
You’re the one drawing an equivalency between USA and China where there is none
Everybody works for someone.
I reject that framing
>I wouldn't touch anything US-based with 10m long pole.
And yet, here you are...
Just when you think it's only the evil dictatorial regime trying to break up NATO along comes a helpful, presumably USA-based HN commenter, to remind you that a lot of USA citizens are also supportive of destroying any semblance of accord with former allied nations and spreading disharmony wherever they go.
It's right for all of us to consider whether our online presence supports fascist dictators be they from USA, Russia, Venezuela, or wherever.
Thanks for the reminder that even here people are sometimes shit, I guess.
> And yet, here you are...
I'm pretty sure you know what the parent poster meant, and you should take it as a compliment to you and our HN community that they didn't intend you, or us, to be included in that definition of 'anything US-based'.
Stirring the pot like that isn't helpful to you, to HN itself, or any of our community.
Hackers -the ones from MIT- and smart people hate the US goverment snooping and Echelon like projects more than Europeans. These were either privacy first since forever. If not, they fought hard to say whatever they liked without consequences.
In current times, encrypted accesses to Usenet and IRC via I2P and the like will boost the platforms more than ever. Why? IRC and Usenet are dead simple, Emacs has ports to everything and among being a Lisp env, an editor and a minimal web browser, it's an IRC and Usenet client too even under Android. Oh, and you can set I2P under Android too. Thus, you just have to set Emacs against it. There are several guides online.
Rocksolid BBS' federate with the whole Usenet (and some newsgroups catch anything text based, no binaries). On I2P proxies to Libera.chat, it's just a matter of time to exist. Meanwhile, there's ILITA IRC with I2PD.
Difficult? A Chinese Bluetooth keyboard it's worth very little today, and the gains are enormous. You can chat with people with really small bandwidths but encrypted either with TLS or I2P. You don't have blocks (except bans under IRC), comment limits, enforced timelines and any enshittification coming from social networks. Also, you can short Usenet threads by score. That's it, it's there a brilliant poster and comp.misc, you set the score for all his comment to 1000; then that random Joe will always be on top in any group. Try that with X/Twitter, Reddit or whatever.
You will be able to chat with Western Europeans, Eastern Europeans, the Japanese, anyone. Forget tribes, forget the bullshit made to earn zillions of cash from X with shitty fabricated polemics. Forget Meta's snooping and industrial stealing. You aren't enforced to give your real name and address.
It is perfectly normal for an anycast network operator to have multiple sites from which they make BGP announcements (which is how anycast works in the first place), which gives them multiple vantage points for this sort of analysis.
Other CDN companies can do it too, it's just that they don't work on signalling their engineering focused organization.
This was a while ago but I think it was Akamai that pioneered that. I remember how impressive it was in the early oughts though we take it for granted now.
You don't have to be cloudflare for this kind of analysis you can do it yourself without even needing an ASN using RIPE RIS.
https://www.ripe.net/analyse/internet-measurements/routing-i...
They have a lot of resources, Cloudflare Is Awesome
> Cloudflare Is Awesome
Until their systems block you for no reason. I recently had a similar issue on a work related site. Fortunately, I was able to reach to the administrator (which is on another country) and had the knowledge to write a report which was useful enough for the said administrator.
And this is for a system which has the same static IP which is not shared with anything for 10ish years.
I recently, with great reluctance, had to put a personal site behind Cloudflare free option. It gets lots of use, but brings no revenue (costs me to run) and I have little spare time.
Found out that I was blocked from it in my default setup. Firefox with default settings, and no VPN.
I'm working hard to turn Cloudflare off.
Cloudflare is not remotely awsome. It's also a solution to a problem (aggressive scrapers that produce DOS) which is worse.
What malicious usage are you seeing? You might have a lot more usage than me, but fail2ban has been enough
Just very high usage all of a sudden, after years of reasonable usage. Google has indexed it (respectfully) since 2008 just fine.
New traffic isn't humans. I blocked some AI scraper user-agents, which helped, a bit. But most new user agents are identifying as vanilla browsers, not scrapers.
I don't have numbers. It was enough to consume all nginx worker_connections. Raising the number doesn't help, as it's just reverse proxying to JVM.
After the switch, Cloudflare showed USA and Singapore as heavy traffic sources.
I don't mind scrapers on the site, but app is a search engine (of sorts) so every page view consumes some CPU. Including 'facet this search' buttons. My (WIP) solution is to rewrite to make it all client-side and put it all on a CDN.
> The user agents are vanilla browsers, not identifying as scrapers.
This is how they get you, alongside with "residential proxy" services they use. They appear to be benign browsers from various homes.
> 10ish years
Systems get infected, and new "residential proxies" get made of unsuspecting internet subscribers all the time.
It's just another IP to them.
The machine in question is not a Windows installation connected to a home router.
The post mentions a number of times that leaks happen "all the time", but the only comparative data shown related to this is for historical leaks from AS8048.
Does anyone have data on what the general frequency of these leaks is likely to be across the network?
I’ve seen leaks impact my company directly 4 or 5 times in 4 years, so I would think often enough since we own a /9~ and don’t change our routes too often.
BGP is outside of my skillset, and I'm sure the analysis is fair and accurate. However, had billion dollar US based company Cloudflare detected widespread manipulation of routing tables by the US secret services, I certainly wouldn't trust them to publish it.
MANERS has some reporting here
https://observatory.manrs.org/#/overview
And Cloud flare has some publicly available reporting in radar
https://radar.cloudflare.com/routing
This is a good opportunity to assess what parts of your own online activity could be impacted by an attacker in the middle (assisted by a BGP leak or otherwise) and, if you're a service provider, how you can protect your customers.
At first pass you probably use HTTPS/TLS for the web, and you know that you shouldn't click through invalid certificate warnings. So the web, tentatively, looks pretty safe.
Email jumps out as vulnerable to eavesdropping, as we largely use opportunistic encryption when transferring messages between mail servers and an on-network-path attacker can use STARTTLS stripping or similar techniques. Most mail servers happily send using cleartext or without validating the TLS certificate. Check that you and your counter-parties are using DNSSEC+DANE, or MTA-STS to ensure that authenticated encryption is always used. Adoption is still quite low, but it's a great time to get started. Watch out for transactional email, like password reset messages, which virtually never validate encryption in transit (https://alexsci.com/blog/is-email-confidential-in-transit-ye... ; instead use multi-factor encryption).
TLS certificates themselves are at risk, unfortunately. An attacker who controls the network in-and-out of your DNS servers can issue domain-verified certificates for your domain; even removing protections like CAA records. DNSSEC is the classic solution here, although using a geographically distributed DNS provider should also work (see multi-perspective validation). Certificate transparency log monitoring should detect any attacker-issued certificates (a review of certificates issued for .ve domains would be interesting).
Ideally, we should build an internet where we don't need to trust the network layer. A BGP route leak would be a performance/availability concern only. We're not there yet, but now is a great time to take the next step in that direction.
As someone who knows nothing about networking, this felt really easy to follow. Thanks for sharing!
Slightly off topic, but if I want to understand the concepts discussed in this article, what all topics should I learn? Is this a good starting place or enough to understand everything in this article - https://beej.us/guide/bgnet/
Because of the formulation of the question, I assume (please don't be irritated, if wrong) that you have very little knowledge of networking. In that case, it won't harm the Beej guide, but probably not the best point to start. The article discuses BGP protocol, which is totally absent in the linked guide. You may write literally millions of networking applications without need to know anything about BGP. Only if you are working in the backbone of the internet, you will encounter BGP, not even in big private networks. It will be a long way to really start from 0 up to BGP.
I was trying to think of an analogy
Kind of like wanting to learn how a car engine works and asking about fleet management in trucks/ lorries.
BGP is one of things I've learnt then forget the next day (multiple times)
Me too. The problem is that you just (at least I) never use day to day.
I have to start from somewhere. I don’t mind the grind. I am fascinated by networking. Will begin with beej guide, and go from there.
This is what helped me back in the day when I did that stuff. I didn't know my ass from a hole in the ground and Avi's guides got me up and running, fully multi-homed!
http://freedman.net/bgbgp.ppt
http://freedman.net/bgp102.ppt
http://freedman.net/multi.ppt
http://avi.freedman.net/fromnetaxs/multi.html
http://avi.freedman.net/fromnetaxs/bgp/bgp.html
http://freedman.net/choose.ppt
First learn the basics of IP networking, looking for CCNA certification materials or Comptia Network+ materials would give you a good base of knowledge.
BGP is just the way networks exchange routes (paths to a specific IP address) between each other. Once you know a little bit about how routing works it will make sense.
BGP itself is fairly simple to understand and entirely disconnected from day to day networking, you could start on it with very minimal understanding of anything else.
What about this? https://www.cloudflare.com/en-au/learning/security/glossary/...
The goal for me is conceptual understanding, not to build an ISP. And network knowledge is light... just enough to be a cloud monkey
The real question tho is: how would you become the mitm reserving for yourself the benefit of the doubt?
Related initial aftermath:
There were BGP anomalies during the Venezuela blackout
https://news.ycombinator.com/item?id=46504963
I probably glossed over it in all the posts but…
What is a BGP?
It's the protocol used by carriers to route traffic globally. (Make automated decisions about which core router should receive the traffic coming out of your AS ("network", kinda) )
It's entirely detached from anything else so you're pretty unlikely to have heard of it. In that way it's similar to SS7.
The Internet is a network of Networks, BGP (Boarder Gateway Protocol) is how routers tell other routers what networks they are connected to. This allows you to connect to any device on the Internet, even if you have to go through 5 different networks to get there.
There have been BGP shenanigans before.
https://arstechnica.com/information-technology/2018/11/major...
> Google goes down after major BGP mishap routes traffic through China
> 8 min read
hah.
> As news unfolds surrounding the U.S. capture and arrest of Venezuelan leader Nicolás Maduro, ... It is also noteworthy that these leak events begin over twelve hours prior to the U.S. military strikes in Venezuela. ...
This is how I imagine Russian companies in Russia write about the Russian war on Ukraine.