Glad that it's published, I'd been following it since ESNI draft days. Was pretty useful back when I was in India since Jio randomly blocked websites, and cloudflare adopted the ESNI draft on its servers as did Firefox client side which made their SNI based blocking easy to bypass.
There was a period where I think both disabled ESNI support as work was made on ECH, which now is pretty far along. I was even able to setup a forked nginx w/ ECH support to build a client(browser) tester[0].
Hopefully now ECH can get more mainstream in HTTPS servers allowing for some fun configs.
A pretty interesting feature of ECH is that the server does not need to validate the public name (it MAY) , so clients can use public_name's that middleboxes (read: censors) approve to connect to other websites. I'm trying to get this added to the RustTLS client[1], now might be a good time to pick that back up.
If i'm not mistaken its because IPs are actually much easier to rotate than domains.
E.g. all the users will remember `example.com` , underlying it doesn't matter what IP it resolves to. If the IP gets "burned" , then the providers can rotate to a new IP (if their provider allows).
Vs. telling your users to use a new domain `example.org` , fake websites etc.
Also sensible ISPs usually don't block IPs since for services behind a CDN it could lead to other websites being blocked, though of course sometimes this is ignored. See also: https://blog.cloudflare.com/consequences-of-ip-blocking/
> Was pretty useful back when I was in India since Jio randomly blocked websites
With Jio, you don't really need ECH at all. The blocks are mostly rudimentary and bypassed with encrypted DNS (DoH / DoT / DNSCrypt) and Firefox (which fragments the TLS ClientHello packets into two).
Should've added this was back in like 2018 or so. Setting up DoH was harder than enabling SNI, and from my testing back then they were hard filtering on SNI (e.g. I used OpenSSL CLI to set the SNI to `pornhub.com` and connect to "known good" IPs, it'd still get reset).
Funnily enough, not setting the SNI and connecting the the origin IP, and then requesting the page worked fine.
one angle that hasn't come up here yet: ECH basically kills TLS fingerprinting as a bot detection signal
right now tools like Cloudflare Bot Management rely heavily on JA3/JA4 hashes - they fingerprint the ClientHello to identify scrapers vs real browsers. if the ClientHello is encrypted, that whole detection layer collapses. you can still do behavioral analysis and JS challenges, but the pre-HTTP layer that currently catches a huge chunk of naive bots - gone
curious how Cloudflare handles this internally given they're one of the biggest ECH adopters but also one of the biggest bot detection vendors. seems like they're eating their own lunch on this one, or they've already shifted their detection stack to not rely on it as much
Cloudflare can and must decrypt the ClientHello for the sites it serves in order to actually serve the traffic. Using ECH with CF means you use their ECH domain and their keys.
If you control the domain you're fingerprinting clients on, you can decrypt the inner ClientHello and fingerprint on that.
If you're not in control of the domain you're fingerprinting, then ECH is working as intended.
I don't expect naive bots to implement ECH any time soon, though. If a bot can't be bothered to download curl-impersonate, they won't pass any ECH flags either.
I wrote about ECH a couple of months ago, when the specs were still in draft but already approved for publication. It's a short read, if you're not already familiar with ECH and its history:
https://www.feistyduck.com/newsletter/issue_127_encrypted_cl...
In addition to the main RFC 9849, there is also RFC 9848 - "Bootstrapping TLS Encrypted ClientHello with DNS Service Bindings": https://datatracker.ietf.org/doc/rfc9848/
There's an example of how it's used in the article.
Thanks for the writeup, Ivan, I am a great fan of your work!
Now we need to get Qualys to cap SSL Labs ratings at B for servers that don't support ECH. Also those that don't have HSTS and HSTS Preload while we're at it.
Thanks! Sadly, SSL Labs doesn't appear to be actively maintained. I've noticed increasing gaps in its coverage and inspection quality. I left quite a while ago (2016) and can't influence its grading any more, sadly.
(Disclosure: I'm a Caddy maintainer), Caddy already supports ECH, leaning on the DNS plugins to automate setting the DNS HTTPS records to wire it up. Here's a lot of technical detail about it https://caddyserver.com/docs/automatic-https#encrypted-clien...
> In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity [RFC9525]. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. Section 7.4 of [RFC3986] and [WHATWG-IPV4]) MUST reject names that would be interpreted as IPv4 addresses.
Aside from apparently not considering the existence of IPv6, why are IP-based certificates explicitly ruled out? This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
\> This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
Actually you can setup ECH on your server, and configure the public_name to be something like `cloudflare-ech.com` , so clients would indeed use that in the OuterSNI, connect to you, without you needing to use CF. And middleboxes might think they are indeed connecting to CF (though CF publishes their IP ranges so this could be checked elsewhere).
IPv6 addresses aren't confusable with a DNS name in these syntaxes AFAIU so it's not that they didn't consider IPv6 but that it's not relevant to the issue.
Yes, "Don't stand out" technologies like ECH aren't useful if you inherently stand out anyway. They're intended to make broad surveillance and similar undirected attacks less effective, they aren't magic invisibility cloaks and won't protect you if you're a singular target.
I think it's saying that you can't make the name look like an IP address; i.e. if the syntax were www.google.com[142.250.117.139] (I'm making this syntax up) you couldn't put 142.250.117.139[142.250.117.139].
The syntax being referred to includes some obscure, outdated addressing formats (IPv4 addresses represented as two or three number groups in dotted notation rather than the normal 4).
However, "DNS-based reference identity [RFC9525]" seems to explicitly disallow IP-based certificates by requiring a DNS name. I can only interpret the sentence I quoted as written to say "make sure you never ever accidentally validate an IP address".
The colon isn’t a valid character in DNS, so there’s just no risk of confusing IPv6 addresses (which contain at least one colon in all notations I’ve seen).
For IPv4, there’s room for ambiguity.
And how are IP certificates required for small servers?
I can't think of a single numeric TLD, so I don't think anyone is confusing IP literals with domain names, unless they're doing so extremely lazily.
> And how are IP certificates required for small servers?
You need a valid certificate as the outer certificate which contains an SNI that will still be readable. For cloudflare.com and google.com that's easy; you can't tell what website Cloudflare is proxying and whether Google is serving you Youtube, Gmail, or Google Search content.
For an independently-hosted myhumanrightsblog.net, that's not as easy. They'd need another domain reachable on that server to set up the ECH connection to hide the risky TLD. Clients being snooped on still get specific domains logged.
IP certificates work around that issue by validating the security of the underlying connection rather than any specific hostname. Any server could be serving any hostname over an IP-address-validated connection. For snooped-on clients, the IP address is already part of the network traffic anyway, but no domains ever hit the traffic logs at all.
ECH doesn't benefit you if you're connecting directly to one IP. Middleboxes can track that you're connecting to this IP.
ECH prevents tracking through routing layers where your ClientHello might contain foo.example.com or bar.example.com but route via the same IP (Cloudflare). A middlebox can see you are using a cloudflare hosted website, but not know what cloudflare website.
There's no benefit encrypting the SNI with 10.20.30.40 if they can see you're connecting to 10
20.30.40 anyway
It's an interesting feature, but it's pushing my buttons lately. Specifically on Cloudflare it is on by default, and on the free tier you can't disable it and you need a business plan for it. Which I think is stupid, but never the less it's causing us problems. We are trying some split-dns setup for a company "intranet", and if the site's have be accessed before, the ECH is remembered. So the browser tries and it eventually fails with ECH Error or on Firefox it just hangs like it's loading all the time. And it's so frustrating, because sometimes it works, sometimes it doesn't, you can clear cache and stuff and it still won't work, sometimes it works in Incognito sometimes it doesn't. This is not a real problem, but since we haven't fully switched to the "intranet" and we use some of the WAF features of Cloudflare sometimes it is so frustrating.
My understanding is that you can use split mode to only have the load balancer decrypt the server name section, and forward the actual session and key exchange down to the backend without doing double layer encryption.
Only if the attacker has a valid certificate for the domain to complete the handshake with.
Relying on HTTPS and SVCB records will probably allow a downgrade for some attackers, but if browsers roll out something akin to the HSTS preload list, then downgrade attacks become pretty difficult.
DNSSEC can also protect against malicious SVCB/HTTPS records and the spec recommends DoT/DoH against local MitM attacks to prevent this.
Glad that it's published, I'd been following it since ESNI draft days. Was pretty useful back when I was in India since Jio randomly blocked websites, and cloudflare adopted the ESNI draft on its servers as did Firefox client side which made their SNI based blocking easy to bypass.
There was a period where I think both disabled ESNI support as work was made on ECH, which now is pretty far along. I was even able to setup a forked nginx w/ ECH support to build a client(browser) tester[0].
Hopefully now ECH can get more mainstream in HTTPS servers allowing for some fun configs.
A pretty interesting feature of ECH is that the server does not need to validate the public name (it MAY) , so clients can use public_name's that middleboxes (read: censors) approve to connect to other websites. I'm trying to get this added to the RustTLS client[1], now might be a good time to pick that back up.
[0] https://rfc9849.mywaifu.best:3443/ [1] https://github.com/rustls/rustls/issues/2741
Why didn't the Indian government block traffics based on IP instead? That would make it much harder to bypass.
If i'm not mistaken its because IPs are actually much easier to rotate than domains.
E.g. all the users will remember `example.com` , underlying it doesn't matter what IP it resolves to. If the IP gets "burned" , then the providers can rotate to a new IP (if their provider allows).
Vs. telling your users to use a new domain `example.org` , fake websites etc.
Also sensible ISPs usually don't block IPs since for services behind a CDN it could lead to other websites being blocked, though of course sometimes this is ignored. See also: https://blog.cloudflare.com/consequences-of-ip-blocking/
> Was pretty useful back when I was in India since Jio randomly blocked websites
With Jio, you don't really need ECH at all. The blocks are mostly rudimentary and bypassed with encrypted DNS (DoH / DoT / DNSCrypt) and Firefox (which fragments the TLS ClientHello packets into two).
Also: https://news.ycombinator.com/item?id=34232190
Should've added this was back in like 2018 or so. Setting up DoH was harder than enabling SNI, and from my testing back then they were hard filtering on SNI (e.g. I used OpenSSL CLI to set the SNI to `pornhub.com` and connect to "known good" IPs, it'd still get reset).
Funnily enough, not setting the SNI and connecting the the origin IP, and then requesting the page worked fine.
one angle that hasn't come up here yet: ECH basically kills TLS fingerprinting as a bot detection signal
right now tools like Cloudflare Bot Management rely heavily on JA3/JA4 hashes - they fingerprint the ClientHello to identify scrapers vs real browsers. if the ClientHello is encrypted, that whole detection layer collapses. you can still do behavioral analysis and JS challenges, but the pre-HTTP layer that currently catches a huge chunk of naive bots - gone
curious how Cloudflare handles this internally given they're one of the biggest ECH adopters but also one of the biggest bot detection vendors. seems like they're eating their own lunch on this one, or they've already shifted their detection stack to not rely on it as much
Cloudflare can and must decrypt the ClientHello for the sites it serves in order to actually serve the traffic. Using ECH with CF means you use their ECH domain and their keys.
If you control the domain you're fingerprinting clients on, you can decrypt the inner ClientHello and fingerprint on that.
If you're not in control of the domain you're fingerprinting, then ECH is working as intended.
I don't expect naive bots to implement ECH any time soon, though. If a bot can't be bothered to download curl-impersonate, they won't pass any ECH flags either.
It doesn't prevent fingerprinting, stop spreading misinformation. It only prevents your ISP from knowing what website you're connecting to.
Since most ISPs also maintain their own DNS resolver, they could always reverse lookup the IP address AFAIK.
True. ECH is useless if you're using plain DNS. DNS over TLS or HTTPS is the way to go.
What OP wrote seems correct:
> ECH basically kills TLS fingerprinting as a bot detection signal
They are not talking about fingerprinting in general. Please elaborate how else TLS fingerprinting can be done.
I am talking about TLS fingerprinting, not JS fingerprinting.
> Please elaborate how else TLS fingerprinting can be done.
By doing everything as it is right now?
How would you (an arbitrary web server) fingerprint a TLS connection if the Client Hello is encrypted?
By decrypting it? I don't think you know how TLS, or E2E works in general. ISP doesn't perform the fingerprinting, the server does.
Of course! My bad, thanks for engaging.
The website owner (or cloudflare in this case) has the keys to decrypt the client hello. That's necessary for routing information.
You're right, sorry! I got confused myself.
I wrote about ECH a couple of months ago, when the specs were still in draft but already approved for publication. It's a short read, if you're not already familiar with ECH and its history: https://www.feistyduck.com/newsletter/issue_127_encrypted_cl...
In addition to the main RFC 9849, there is also RFC 9848 - "Bootstrapping TLS Encrypted ClientHello with DNS Service Bindings": https://datatracker.ietf.org/doc/rfc9848/
There's an example of how it's used in the article.
Thanks for the writeup, Ivan, I am a great fan of your work!
Now we need to get Qualys to cap SSL Labs ratings at B for servers that don't support ECH. Also those that don't have HSTS and HSTS Preload while we're at it.
Thanks! Sadly, SSL Labs doesn't appear to be actively maintained. I've noticed increasing gaps in its coverage and inspection quality. I left quite a while ago (2016) and can't influence its grading any more, sadly.
> There's an example of how it's used in the article
A bit tricky in Go, but nothing too complicated. We implemented ECH in Aug 2024 for our DNS Android app and it has worked nicely since: https://github.com/celzero/firestack/blob/09b26631a2eac2cf9c...
(Disclosure: I'm a Caddy maintainer), Caddy already supports ECH, leaning on the DNS plugins to automate setting the DNS HTTPS records to wire it up. Here's a lot of technical detail about it https://caddyserver.com/docs/automatic-https#encrypted-clien...
Something that puzzles me about ECH:
> In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity [RFC9525]. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. Section 7.4 of [RFC3986] and [WHATWG-IPV4]) MUST reject names that would be interpreted as IPv4 addresses.
Aside from apparently not considering the existence of IPv6, why are IP-based certificates explicitly ruled out? This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
\> This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
Actually you can setup ECH on your server, and configure the public_name to be something like `cloudflare-ech.com` , so clients would indeed use that in the OuterSNI, connect to you, without you needing to use CF. And middleboxes might think they are indeed connecting to CF (though CF publishes their IP ranges so this could be checked elsewhere).
IPv6 addresses aren't confusable with a DNS name in these syntaxes AFAIU so it's not that they didn't consider IPv6 but that it's not relevant to the issue.
Yes, "Don't stand out" technologies like ECH aren't useful if you inherently stand out anyway. They're intended to make broad surveillance and similar undirected attacks less effective, they aren't magic invisibility cloaks and won't protect you if you're a singular target.
I think it's saying that you can't make the name look like an IP address; i.e. if the syntax were www.google.com[142.250.117.139] (I'm making this syntax up) you couldn't put 142.250.117.139[142.250.117.139].
The syntax being referred to includes some obscure, outdated addressing formats (IPv4 addresses represented as two or three number groups in dotted notation rather than the normal 4).
However, "DNS-based reference identity [RFC9525]" seems to explicitly disallow IP-based certificates by requiring a DNS name. I can only interpret the sentence I quoted as written to say "make sure you never ever accidentally validate an IP address".
I don't think your interpretation is right. If it were,
> Clients that incorporate DNS names and IP addresses into the same syntax
They wouldn't mention the IP addresses at all. Also, notice the word "and".
The colon isn’t a valid character in DNS, so there’s just no risk of confusing IPv6 addresses (which contain at least one colon in all notations I’ve seen).
For IPv4, there’s room for ambiguity.
And how are IP certificates required for small servers?
> For IPv4, there’s room for ambiguity.
I can't think of a single numeric TLD, so I don't think anyone is confusing IP literals with domain names, unless they're doing so extremely lazily.
> And how are IP certificates required for small servers?
You need a valid certificate as the outer certificate which contains an SNI that will still be readable. For cloudflare.com and google.com that's easy; you can't tell what website Cloudflare is proxying and whether Google is serving you Youtube, Gmail, or Google Search content.
For an independently-hosted myhumanrightsblog.net, that's not as easy. They'd need another domain reachable on that server to set up the ECH connection to hide the risky TLD. Clients being snooped on still get specific domains logged.
IP certificates work around that issue by validating the security of the underlying connection rather than any specific hostname. Any server could be serving any hostname over an IP-address-validated connection. For snooped-on clients, the IP address is already part of the network traffic anyway, but no domains ever hit the traffic logs at all.
ECH doesn't benefit you if you're connecting directly to one IP. Middleboxes can track that you're connecting to this IP.
ECH prevents tracking through routing layers where your ClientHello might contain foo.example.com or bar.example.com but route via the same IP (Cloudflare). A middlebox can see you are using a cloudflare hosted website, but not know what cloudflare website.
There's no benefit encrypting the SNI with 10.20.30.40 if they can see you're connecting to 10 20.30.40 anyway
It's an interesting feature, but it's pushing my buttons lately. Specifically on Cloudflare it is on by default, and on the free tier you can't disable it and you need a business plan for it. Which I think is stupid, but never the less it's causing us problems. We are trying some split-dns setup for a company "intranet", and if the site's have be accessed before, the ECH is remembered. So the browser tries and it eventually fails with ECH Error or on Firefox it just hangs like it's loading all the time. And it's so frustrating, because sometimes it works, sometimes it doesn't, you can clear cache and stuff and it still won't work, sometimes it works in Incognito sometimes it doesn't. This is not a real problem, but since we haven't fully switched to the "intranet" and we use some of the WAF features of Cloudflare sometimes it is so frustrating.
Split dns is such an ugly hack, it causes no end of hard to debug problems.
It is yeah, but it is a temporary solution while we are working out the setup.
Will this have an impact on Loadbalancers? Like does one have to do client side load balancing like in gRPC?
My understanding is that you can use split mode to only have the load balancer decrypt the server name section, and forward the actual session and key exchange down to the backend without doing double layer encryption.
The loadbalancer can force a downgrade .
If the load balancer can force a downgrade, an attacker can do it as well.
Only if the attacker has a valid certificate for the domain to complete the handshake with.
Relying on HTTPS and SVCB records will probably allow a downgrade for some attackers, but if browsers roll out something akin to the HSTS preload list, then downgrade attacks become pretty difficult.
DNSSEC can also protect against malicious SVCB/HTTPS records and the spec recommends DoT/DoH against local MitM attacks to prevent this.