Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.
I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article.
The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.
But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for not only upgrading your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.
Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).
I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.
> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions
Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.
One year ago I might agree that Kubernetes is an overkill but today?
Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again.
One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
> One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Using Kubernetes because you're unable to grok docker's networking enough so you can't run multiple containers using their own ports and not conflicting with other stuff sounds like a recipe for disaster, even (especially?) if you use agents for this. Particularly if you let them manage a production environment, you're bound to lose important data eventually.
Even as a solo dev there's generally been a yawning gap between k8s and manual infra that nothing has ever filled that well and it's part of why things like Heroku were so popular for a while.
Well, maybe especially as a solo dev. Things like Heroku and other tools that were largely called PaaS at the time were very popular with individuals and small teams but they had limitations for enterprise development--and even ran into barriers once anyone ran into those limitations.
This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board.
It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.
Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.
Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"
> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won.
>
> My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know.
Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.
To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers.
Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.
And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.
Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.
If you struggle with any of that (a lot of what you listed is not strictly necessary to running managed kubernetes, specifically EKS) you are also going to struggle with a lot of other things on AWS, or wrangling a VM setup at any kind of scale.
A pretty nothing burger of a post with a bunch of ai-isms. Is this written by a real human?
K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.
Honestly I felt the same way for a while but the more I'm exposed to both Fortune 500 companies and ones who have a handful of employees I see Kubernetes as just a good starting point rather than adopting it later.
It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly.
That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.
Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.
I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article.
The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.
But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for not only upgrading your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.
Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).
I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.
> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions
Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.
One year ago I might agree that Kubernetes is an overkill but today?
Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again.
One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
> One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Using Kubernetes because you're unable to grok docker's networking enough so you can't run multiple containers using their own ports and not conflicting with other stuff sounds like a recipe for disaster, even (especially?) if you use agents for this. Particularly if you let them manage a production environment, you're bound to lose important data eventually.
> pretty much free lunch.
Aah, famous last words of the young :)
Interesting. I have just started reading about Kubernetes. Is there an reading material that goes over this process you just described?
Even as a solo dev there's generally been a yawning gap between k8s and manual infra that nothing has ever filled that well and it's part of why things like Heroku were so popular for a while.
Well, maybe especially as a solo dev. Things like Heroku and other tools that were largely called PaaS at the time were very popular with individuals and small teams but they had limitations for enterprise development--and even ran into barriers once anyone ran into those limitations.
This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board.
It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.
> First one was uniformity. Every service deploys the same way.
My current company makes this claim, but it's not true. They also have serverless apps, and also have some services running directly on EC2.
They just think of the Kubernetes deployments as the "standard" way.
> Second was shared, hireable knowledge. K8s is basically a lingua franca now.
People were demanding experience with Kubernetes, long before it was reasonable to expect it. Everyone added it to their resume, because they had to.
Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.
I thought that was the point of the article, right?
That the tech benefits may not be there, but they’re using it for the non-tech benefits
Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"
> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know.
Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.
To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers.
Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.
And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.
Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.
If you struggle with any of that (a lot of what you listed is not strictly necessary to running managed kubernetes, specifically EKS) you are also going to struggle with a lot of other things on AWS, or wrangling a VM setup at any kind of scale.
A pretty nothing burger of a post with a bunch of ai-isms. Is this written by a real human?
K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.
Honestly I felt the same way for a while but the more I'm exposed to both Fortune 500 companies and ones who have a handful of employees I see Kubernetes as just a good starting point rather than adopting it later.
It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly.
That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.