Let's Encrypt has been working normally for most of the day. There was a ~90 minute period during which some of our users would have received a higher error rate due to upstream networking issues, but the majority of requests were successful even during that period.
It seems our status.io notes are being misinterpreted as much more severe than they were intended to reflect.
I'm not sure if your higher error rate is sticky per user or something, but I've tried 10+ times throughout the day and have had 0 successes. They all come back as internal server error. That's why I eventually posted.
It would not have been sticky for the entire day. If it was sticky at all, it would have been only during the 90 minute period I referenced. It's most likely that there is some other issue with how you're requesting the cert. Folks can help debug at: https://community.letsencrypt.org/
That explains why one of my IoT vendors is using an expired certificate.
I wish Firefox would just give a mild warning for a recently expired certificate, instead of treating it the same as a true man-in-the-middle attach. It's not like someone who couldn't factor the private key in 200 days could in 201 days or even 300 days.
I'm convinced that we'd have better security, if we didn't have so much security theater. You'd think TLS is useless, from the warning my phone gives if I connected to a public Wi-Fi AP, but then again there's nothing in TLS (or WPA) that prevents it from being used in a way that is completely useless: https://www.youtube.com/watch?v=M1si1y5lvkk
> That explains why one of my IoT vendors is using an expired certificate.
I don't think so. There was a dip in success rates for 90 minutes today, but nobody should be renewing their certificate within 90 minutes of expiration. If you're at that point, something went wrong weeks ago.
"nobody should be renewing their certificate within 90 minutes of expiration"
You obviously haven't worked with hardware guys.
"I mean, what's the point of those last 30 days if you need to renew it 30 days before expiration? Why not just renew it before it expires? If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?"
Mostly 90 days, and we recommend renewing at 60 days for 90 day certs. That gives more than four weeks of leeway.
If you're one of the few early adopters of short-lived (6-day) certs you should renew at 3 days, giving you 3 days for a successful renewal. A 90 minute outage, even if it was a full outage, would not interfere with a successful renewal.
'I'm convinced that we'd have better security, if we didn't have so much security theater."
Can't have security without control
When someone else is in control, it's theater
Because there is no security from that other party and the user's security against others is within that other party's control, not the user's
As some folks have realised, e.g., many HN users, the other party can turn the "security" against the user, locking the user out of their own computer or access to public information, for example
The Fortune 500 and many other companies use "MiTM" to create security, monitoring traffic from their networks, not to compromise it
But it's only the extreme warning that alerts the website (usually via a customer complaining) that the cert hasn't been renewed. Having the lesser warning just kicks the can down the road.
The IoT should have updated the certs weeks in advance. If they haven't done it by day 0 then their process is broken and delaying the scary warning to say day +5 won't solve anything.
Let's Encrypt is operating normally. If you're having trouble, please post the details on the community forum so that folks can help you out. There is external monitoring in place.
I use acme.sh for certs on my personal server and was a little surprised when it started using ZeroSSL by default. Despite being more "corporate" I decided to roll with it and it's worked just fine.
None. Big tech intentionally made Let's Encrypt a single point of giant failure.
> And in case none exists, what does it take to build one?
A new Internet and Web standards stack. The whole problem is self-imposed -- we could have published self-signed Ed25519 keys on the DNS instead, and the result would be more secure than whatever it is we have now.
The banner's colour is based on the "Incident Status;" it's green because services are currently operational. It would be yellow or red if the impact were more severe.
You are getting down-voted for this, which I think is a bit unfair. (I expect I'll get the same.)
Although you don't expand your thesis, as a general feeling, I agree. But, to be fair, it has always been thus, and it has been this way in every forum ever.
I'm old enough to remember the irony in "I read about it on the internet so it must be true" statements, which have existed since the internet was News (NNTP) not web.
In truth, any time you get a random group of people together, of different ages and backgrounds, all of whom self-describe as "smart" you're going to get a lot of chaff mixed in with the wheat.
To some extent you need to simply ignore the nonsense. There's plenty of it and "correcting people who are wrong" is seldom received well.
Let's Encrypt has been working normally for most of the day. There was a ~90 minute period during which some of our users would have received a higher error rate due to upstream networking issues, but the majority of requests were successful even during that period.
It seems our status.io notes are being misinterpreted as much more severe than they were intended to reflect.
I'm not sure if your higher error rate is sticky per user or something, but I've tried 10+ times throughout the day and have had 0 successes. They all come back as internal server error. That's why I eventually posted.
It would not have been sticky for the entire day. If it was sticky at all, it would have been only during the 90 minute period I referenced. It's most likely that there is some other issue with how you're requesting the cert. Folks can help debug at: https://community.letsencrypt.org/
Could it be that he was simply throttled while retrying? That seems plausible, and it would make it seem like a long outage.
I ran the exact same command now and it's working, so it is possible I was unlucky and was hitting all the worst possible cases.
I updated the post title to say (Fixed) now.
Since Let's Encrypt wasn't down most of the day if would be helpful if you could update the title to reflect that.
That explains why one of my IoT vendors is using an expired certificate.
I wish Firefox would just give a mild warning for a recently expired certificate, instead of treating it the same as a true man-in-the-middle attach. It's not like someone who couldn't factor the private key in 200 days could in 201 days or even 300 days.
I'm convinced that we'd have better security, if we didn't have so much security theater. You'd think TLS is useless, from the warning my phone gives if I connected to a public Wi-Fi AP, but then again there's nothing in TLS (or WPA) that prevents it from being used in a way that is completely useless: https://www.youtube.com/watch?v=M1si1y5lvkk
> That explains why one of my IoT vendors is using an expired certificate.
I don't think so. There was a dip in success rates for 90 minutes today, but nobody should be renewing their certificate within 90 minutes of expiration. If you're at that point, something went wrong weeks ago.
"nobody should be renewing their certificate within 90 minutes of expiration"
You obviously haven't worked with hardware guys.
"I mean, what's the point of those last 30 days if you need to renew it 30 days before expiration? Why not just renew it before it expires? If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?"
> weeks ago
How long do you think a certificate lives?
Mostly 90 days, and we recommend renewing at 60 days for 90 day certs. That gives more than four weeks of leeway.
If you're one of the few early adopters of short-lived (6-day) certs you should renew at 3 days, giving you 3 days for a successful renewal. A 90 minute outage, even if it was a full outage, would not interfere with a successful renewal.
'I'm convinced that we'd have better security, if we didn't have so much security theater."
Can't have security without control
When someone else is in control, it's theater
Because there is no security from that other party and the user's security against others is within that other party's control, not the user's
As some folks have realised, e.g., many HN users, the other party can turn the "security" against the user, locking the user out of their own computer or access to public information, for example
The Fortune 500 and many other companies use "MiTM" to create security, monitoring traffic from their networks, not to compromise it
> I wish Firefox would just give a mild warning for a recently expired certificate
Nope, if the SSL industry continues to insist on increasingly short cert lifetimes then I want Firefox to give no quarter when a cert expires.
Play by their rules and fall by their rules too.
How does that help? Seems like mostly the end user suffers.
But it's only the extreme warning that alerts the website (usually via a customer complaining) that the cert hasn't been renewed. Having the lesser warning just kicks the can down the road.
The IoT should have updated the certs weeks in advance. If they haven't done it by day 0 then their process is broken and delaying the scary warning to say day +5 won't solve anything.
omg new tom7!
To be clear, “Degraded Performance” means just that, not “down.” Let’s Encrypt’s issuance is mostly working fine.
I see you are unfamiliar with status page-ese. “Degraded performance” is a term which means some form of “the entire datacenter is probably on fire”.
Although I only post here personally, I work for Let’s Encrypt.
It would be better to say this upfront. I am not blaming you in any way but this would prevent responses such as the parent's (hopefully).
Thanks you for your work!
Let them know that they're having an outage. If their monitors aren't telling them so, they might need to host them off-site.
Let's Encrypt is operating normally. If you're having trouble, please post the details on the community forum so that folks can help you out. There is external monitoring in place.
That would a Microsoft'ese, "Some regions are encountering issues" => "The entire world is down, but our status page is working"
A common confusion; this interpretation only applies to OVH.
ref: https://www.reuters.com/article/world/millions-of-websites-o...
I thought it meant "electricity has ceased to be a physical phenomenon in the general vicinity of our servers"
I have tried many times to renew my certs and have had 0 successes throughout today. It seems to be 100% degraded to me.
That’s unexpected. Please post details on the “Help” topic of the Let’s Encrypt community forum so that folks can take a look.
What % of requests succeeded vs failed? How many certificates were issued during the outage vs the average? That might actually clear things up
Seems not ideal for an entity who seems to be pushing for shorter expiration periods all the time
I think it’s mostly Apple and maybe Google who have the hard-ons for the shortest expiries possible.
To be fair, if someone managed to steal a set of keys to Gmail.com and icloud.com, I would want them to expire as short a time as possible too.
I think revoking them would be better in such a case.
If it goes past 24 hours, that becomes a real worry.
If anyone is renewing certificates with less than a day remaining, that's an issue on their end far more than anything else.
isn't this the other way around ??? because shorter expiration time resulting on more issuing cert and therefore make it more prone to downtime
What are the viable alternatives to LE? And in case none exists, what does it take to build one?
Requirements: free, available to everyone, automation friendly, issues certificates that are actually considered trustworthy by other parties.
ZeroSSL – free 90-day certs via ACME, also has a web UI for cert management
Google Trust Services – free ACME certs, requires a Google account for registration
SSL.com Free DV SSL – offers free 90-day certs through ACME
I use acme.sh for certs on my personal server and was a little surprised when it started using ZeroSSL by default. Despite being more "corporate" I decided to roll with it and it's worked just fine.
Have the EU or Canada pushed to launch an analog of their own?
It seems a bit silly that a service that could be forced by EO to revoke foreign certificates is the backbone of so much of the internet.
This video explores a little on how certificate authorities were given their authority and a lot on how it can fail: https://www.youtube.com/watch?v=M1si1y5lvkk
It's a bit mathy, but if you can make it through that, I highly recommend watching the whole video, especially if you like dad jokes.
Like peers could sign sites?
> What are the viable alternatives to LE?
None. Big tech intentionally made Let's Encrypt a single point of giant failure.
> And in case none exists, what does it take to build one?
A new Internet and Web standards stack. The whole problem is self-imposed -- we could have published self-signed Ed25519 keys on the DNS instead, and the result would be more secure than whatever it is we have now.
I realize this is very much not the point, but the fact that the "Active Incident" banner is green is upsetting.
Their monitors don't seem to be detecting the outage. Sometimes they run directly on the server, and aren't able to detect routing or DNS problems.
The banner's colour is based on the "Incident Status;" it's green because services are currently operational. It would be yellow or red if the impact were more severe.
We're operating normally, but with reduced redundancy. We continue to work with our upstream ISP to identify and resolve the issue.
It's a good thing that acme clients try to renew early, rather than leaving it to the last minute...
thats too bad
:(
The amount of misinformation on this site is astonishing. "Hacker News"..
You are getting down-voted for this, which I think is a bit unfair. (I expect I'll get the same.)
Although you don't expand your thesis, as a general feeling, I agree. But, to be fair, it has always been thus, and it has been this way in every forum ever.
I'm old enough to remember the irony in "I read about it on the internet so it must be true" statements, which have existed since the internet was News (NNTP) not web.
In truth, any time you get a random group of people together, of different ages and backgrounds, all of whom self-describe as "smart" you're going to get a lot of chaff mixed in with the wheat.
To some extent you need to simply ignore the nonsense. There's plenty of it and "correcting people who are wrong" is seldom received well.