> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster.
This is a pretty good definition.
I think part of the challenge is the evolution of K8s over time sometimes makes it feel less like a coherent runtime and more like a pile of glue amalgamated from several different components all stuck together. That and you will have to be aware of how those abstractions stick together with the abstractions from your cloud provider, etc...
It's an application server for multi-part containerized applications, like Tomcat is an application server for applications that can be turned into .war files.
One approach if "dang it, someone/I needed to use kubectl during the outage, how do we get gitops/poor-mans-gitops back in place to match reality" is, either agentically-looping or artisanally-looping, to try simple gitops configurations (or diffs to current gitops configurations) until a dry-run diff with your live configuration results in no changes.
There's more of a spectrum between these than you think, in a way that can be agile for small teams without dedicated investment in gitops. Even with the messes that can occur, I'd take it over the Heroku CLI any day.
I always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh.
I really like the idea of something like Firebase, but it never seems to work out or just move the complexity to the vendor, which is fine, but I like knowing I can roll my own.
In ascending order of functionality and how much complexity you need:
- Docker Compose running on a single server
- Docker Swarm cluster (typically multiple nodes, can be one)
- Hashicorp Nomad or K3s or other light Kubernetes distros
I think K8s couples two concepts: the declarative-style cluster management, and infrastructure + container orchestration. Keep CRDs, remove everything else, and implement the business-specific stuff on top of the CRD-only layer.
This would give something like DBus, except cluster-wide, with declarative features. Then, container orchestration would be an application you install on top of that.
Edit: I see a sibling mentioned KCP. I’ve never heard of it before, but I think that’s probably exactly what I’d like.
Big question is which feature subset you want to replicate.
Kubernetes means everything to everyone. At its core, I think it’s being able to read/write distributed state (which doesn’t need to be etcd) and being able for all the components (especially container hosts) to follow said state. But the ecosystem has expanded significantly beyond that.
Remove abstractions like CNI, CRI, just make these things built-in.
Remove unnecessary things like Ingress, etc, you can always just deploy nginx or whatever reverse proxy directly. Also probably remove persistent volumes, they add a lot of complexity.
Use some automatically working database, not separate etcd installation.
Get rid of control plane. Every node should be both control plane and worker node. Or may be 3 worker nodes should be control plane, whatever, deployer should not think about it.
Add stuff that everyone needs. Centralised log storage, centralised metric scrapping and storage, some simple web UI, central authentication. It's reimplemented in every Kubernetes cluster.
The problem is that it won't be serious enough and people will choose Kubernetes over simpler solutions.
Some people want their k8s logs to be centralized with non k8s logs. Standardizing log storage seems like a challenging problem. Perhaps they could add built in log shipping. But even then, the transfer format needs to be specified.
Adding an idp is pretty standard in k8s... What do you want to actually do different?
Everything in infrastructure is a set of trade-offs that work in both directions.
If you want better monitoring, metrics, availability, orchestration, logging, and so on, you pay for it with time, money, and complexity.
If you can't justify that cost, you're free to use simpler tools.
Just because everyone sets up a Kubernetes / Prometheus / ELK stack to host a web app that would happily run on a single VPS doesn't mean you need to do the same, or that nowadays this is the baseline for running something.
IMO this is what keeps people from building systems that might challenge kubernetes. Everyone wants to say Kuberentes is too complex, so we built something that does much less. I respect that! But I think it usually fails to grok what Kubernetes is and why it's such an interesting and vital rallying point, that's so thoroughly captured our systems-making. Let's look at the premise:
> That’s why I like to think of Kubernetes as a runtime for declarative infrastructure with a type system.
You can go build a simple way to deploy containers or ship apps: but you are missing what I think allows Kubernetes to be such a big tent, thats a core useful platform for so many. Kubernetes works the same for all types, for everything you want to manage. It's the same desired state management + autonomic systems patterns, whatever you are doing. An extensible platform with a very simple common core.
There are other takes and other tries, but managing desired state for any kind of type is a huge win that allows many people to find their own uses for kube, that is absolutely the cornerstone to it's popularity.
If you do want less, the one project I'd point to that is kubernetes without the kubernetes complexity is KCP. It's just the control plane. It doesn't do anything at all. This to me is much simpler. It's not finding a narrowly defined use case to focus on, it's distilling out the general system into it's simplest parts. Rebuilding a good simple bespoke app container launching platform around KCP would be doable, and maintain the overarching principles that make Kube actually interesting.
I seriously think there is something deeply rotten with our striving for simplicity. I know we've all been burned, and there's so often we want to throw up our hands, and I get it. But the way out is through. I'd rather dance the dance & try to scout for better further futures, than reject & try to walk back.
The allure of declarative approaches to complex problem solving has finally been worn down to nothing for me and Kubernetes was the last straw, nearly 10 years ago.
The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else, invariably generates a horror show of extremely verbose boilerplate, duplication, bloat, delays and pain.
If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.
They'll hum along quietly serving many thousands of actual customers and likely cost less to purchase than a single month (or at worst, quarter) of today's cloud-based muggings.
When you pay, you'll pay for bandwidth and that's real value that also happens to make your work environment more efficient.
If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.
You can literally get a Linux box (or two) in the corner and run:
How am I installing a monolith and a database on this Linux box without Kubernetes? Be specific, just show the commands for me to run. Kubernetes that will work for ~anything. HNers spend more tokens complaining about the complexity than it takes to setup.
The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else
Like, brainfuck? Like bash? Like Terraform HCL puppet chef ansible pile-o-scripts? The effort required to output your desired infrastructure's definition as JSON shouldn't really be that gargantuan. You express yourself in anything else but it can't be dumped to JSON?
> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster.
Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach.
Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion - it is super important to treat a container orchestrator a container orchestrator.
I feel like the author has a good grasp of the Kubernetes design... What about the approach is problematic? And why don't you think that is how Kubernetes was designed to be used?
But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s. Or perhaps using non-native tools or wrappers.
> But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s.
Yes, and 99% of the companies do this. It is quite common to use Terraform/AWS CDK/Pulumi/etc to provision the infrastructure, and ArgoCD/Helm/etc to manage the resources on Kubernetes. There is nothing wrong with it.
> the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach
But some more concrete stories:
Once, while I was on call, I got paged because a Kubernetes node was running out of disk space. The root cause was the logging pipeline.
Normally, debugging a "no space left on device" issue in a logging pipeline is fairly straightforward, if the tools are used as intended.
This time, they weren't.
The entire pipeline was managed by a custom-built logging operator, designed to let teams describe logging pipelines declaratively. The problem? The resource definitions alone were around 20,000 lines of YAML. In the middle of the night, I had to reverse-engineer how the operator translated that declarative configuration into an actual pipeline.
It took three days and multiple SREs to fully understand and fix the issue. Without such a declarative magic it takes usually 1 hour to solve such an issue.
Another example: external-dns.
It's commonly used to manage DNS declaratively in Kubernetes. We had multiple clusters using Route 53 in the same AWS account. Route 53 has a global API request limit per account.
When two or more clusters tried to reconcile DNS records at the same time, one would hit the quota. The others would partially fail, drift out of sync, and trigger retries - creating one of the messiest cross-cluster race conditions I've ever dealt with.
Yes, there is a term for a system that handles a declarative state of infrastructure and does reconciliation versus current state; a control plane. We have been talking about control planes in devops/ SRE for a number of years now! Welcome to the conversation.
> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster.
This is a pretty good definition.
I think part of the challenge is the evolution of K8s over time sometimes makes it feel less like a coherent runtime and more like a pile of glue amalgamated from several different components all stuck together. That and you will have to be aware of how those abstractions stick together with the abstractions from your cloud provider, etc...
It's an application server for multi-part containerized applications, like Tomcat is an application server for applications that can be turned into .war files.
One approach if "dang it, someone/I needed to use kubectl during the outage, how do we get gitops/poor-mans-gitops back in place to match reality" is, either agentically-looping or artisanally-looping, to try simple gitops configurations (or diffs to current gitops configurations) until a dry-run diff with your live configuration results in no changes.
For instance, with Helm, I've had success using Helmfile's diffs (which in turn use https://github.com/databus23/helm-diff) to do this.
There's more of a spectrum between these than you think, in a way that can be agile for small teams without dedicated investment in gitops. Even with the messes that can occur, I'd take it over the Heroku CLI any day.
I always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh.
I really like the idea of something like Firebase, but it never seems to work out or just move the complexity to the vendor, which is fine, but I like knowing I can roll my own.
> What's the really simple K8s?
It's k3s. You drop a single binary onto the node, run it, and you have a fully functional one-node k8s cluster.
In ascending order of functionality and how much complexity you need:
> What's the really simple K8s?
I think K8s couples two concepts: the declarative-style cluster management, and infrastructure + container orchestration. Keep CRDs, remove everything else, and implement the business-specific stuff on top of the CRD-only layer.
This would give something like DBus, except cluster-wide, with declarative features. Then, container orchestration would be an application you install on top of that.
Edit: I see a sibling mentioned KCP. I’ve never heard of it before, but I think that’s probably exactly what I’d like.
Big question is which feature subset you want to replicate.
Kubernetes means everything to everyone. At its core, I think it’s being able to read/write distributed state (which doesn’t need to be etcd) and being able for all the components (especially container hosts) to follow said state. But the ecosystem has expanded significantly beyond that.
Of course things can be simpler.
Remove abstractions like CNI, CRI, just make these things built-in.
Remove unnecessary things like Ingress, etc, you can always just deploy nginx or whatever reverse proxy directly. Also probably remove persistent volumes, they add a lot of complexity.
Use some automatically working database, not separate etcd installation.
Get rid of control plane. Every node should be both control plane and worker node. Or may be 3 worker nodes should be control plane, whatever, deployer should not think about it.
Add stuff that everyone needs. Centralised log storage, centralised metric scrapping and storage, some simple web UI, central authentication. It's reimplemented in every Kubernetes cluster.
The problem is that it won't be serious enough and people will choose Kubernetes over simpler solutions.
Some people want their k8s logs to be centralized with non k8s logs. Standardizing log storage seems like a challenging problem. Perhaps they could add built in log shipping. But even then, the transfer format needs to be specified.
Adding an idp is pretty standard in k8s... What do you want to actually do different?
Everything in infrastructure is a set of trade-offs that work in both directions.
If you want better monitoring, metrics, availability, orchestration, logging, and so on, you pay for it with time, money, and complexity.
If you can't justify that cost, you're free to use simpler tools.
Just because everyone sets up a Kubernetes / Prometheus / ELK stack to host a web app that would happily run on a single VPS doesn't mean you need to do the same, or that nowadays this is the baseline for running something.
IMO this is what keeps people from building systems that might challenge kubernetes. Everyone wants to say Kuberentes is too complex, so we built something that does much less. I respect that! But I think it usually fails to grok what Kubernetes is and why it's such an interesting and vital rallying point, that's so thoroughly captured our systems-making. Let's look at the premise:
> That’s why I like to think of Kubernetes as a runtime for declarative infrastructure with a type system.
You can go build a simple way to deploy containers or ship apps: but you are missing what I think allows Kubernetes to be such a big tent, thats a core useful platform for so many. Kubernetes works the same for all types, for everything you want to manage. It's the same desired state management + autonomic systems patterns, whatever you are doing. An extensible platform with a very simple common core.
There are other takes and other tries, but managing desired state for any kind of type is a huge win that allows many people to find their own uses for kube, that is absolutely the cornerstone to it's popularity.
If you do want less, the one project I'd point to that is kubernetes without the kubernetes complexity is KCP. It's just the control plane. It doesn't do anything at all. This to me is much simpler. It's not finding a narrowly defined use case to focus on, it's distilling out the general system into it's simplest parts. Rebuilding a good simple bespoke app container launching platform around KCP would be doable, and maintain the overarching principles that make Kube actually interesting.
I seriously think there is something deeply rotten with our striving for simplicity. I know we've all been burned, and there's so often we want to throw up our hands, and I get it. But the way out is through. I'd rather dance the dance & try to scout for better further futures, than reject & try to walk back.
The allure of declarative approaches to complex problem solving has finally been worn down to nothing for me and Kubernetes was the last straw, nearly 10 years ago.
The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else, invariably generates a horror show of extremely verbose boilerplate, duplication, bloat, delays and pain.
If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.
They'll hum along quietly serving many thousands of actual customers and likely cost less to purchase than a single month (or at worst, quarter) of today's cloud-based muggings.
When you pay, you'll pay for bandwidth and that's real value that also happens to make your work environment more efficient.
If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.
You can literally get a Linux box (or two) in the corner and run:
How am I installing a monolith and a database on this Linux box without Kubernetes? Be specific, just show the commands for me to run. Kubernetes that will work for ~anything. HNers spend more tokens complaining about the complexity than it takes to setup.The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else
Like, brainfuck? Like bash? Like Terraform HCL puppet chef ansible pile-o-scripts? The effort required to output your desired infrastructure's definition as JSON shouldn't really be that gargantuan. You express yourself in anything else but it can't be dumped to JSON?
> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster.
Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach.
Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion - it is super important to treat a container orchestrator a container orchestrator.
I feel like the author has a good grasp of the Kubernetes design... What about the approach is problematic? And why don't you think that is how Kubernetes was designed to be used?
I wrote some personal stories below in this thread as a response to another user.
But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s. Or perhaps using non-native tools or wrappers.
> But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s.
Yes, and 99% of the companies do this. It is quite common to use Terraform/AWS CDK/Pulumi/etc to provision the infrastructure, and ArgoCD/Helm/etc to manage the resources on Kubernetes. There is nothing wrong with it.
It would have helped if you tell us why you don’t like this approach.
It's right there:
> the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach
But some more concrete stories:
Once, while I was on call, I got paged because a Kubernetes node was running out of disk space. The root cause was the logging pipeline. Normally, debugging a "no space left on device" issue in a logging pipeline is fairly straightforward, if the tools are used as intended. This time, they weren't.
The entire pipeline was managed by a custom-built logging operator, designed to let teams describe logging pipelines declaratively. The problem? The resource definitions alone were around 20,000 lines of YAML. In the middle of the night, I had to reverse-engineer how the operator translated that declarative configuration into an actual pipeline. It took three days and multiple SREs to fully understand and fix the issue. Without such a declarative magic it takes usually 1 hour to solve such an issue.
Another example: external-dns. It's commonly used to manage DNS declaratively in Kubernetes. We had multiple clusters using Route 53 in the same AWS account. Route 53 has a global API request limit per account. When two or more clusters tried to reconcile DNS records at the same time, one would hit the quota. The others would partially fail, drift out of sync, and trigger retries - creating one of the messiest cross-cluster race conditions I've ever dealt with.
And I have plenty more stories like these.
Love the HN title mod here lol
HN removes a "how" if the post starts with it, sometimes making it hilarious.
Tbh the missing how is probably why I followed the link. And I appreciated the post, so net positive outcome here.
I’ve seen this a few times now, what’s the context/background on why this is done?
I think maybe they check for "How to [...]" and other variations? "How I broke TLS 1.3" -> "I broke TLS 1.3"
yeah but why?
Over-engineering to solve a problem that doesn't exist, thereby making one.
Otherwise you might confuse it with the HN mod tomhow.
Same reason they remove "dang" if the post starts with it, like the discussion about "Dang! Who ate the middle out of the daddy longlegs".
https://ifunny.co/picture/dang-who-ate-the-middle-out-of-the...
yeah, really wish they would fix that one!
Yes, there is a term for a system that handles a declarative state of infrastructure and does reconciliation versus current state; a control plane. We have been talking about control planes in devops/ SRE for a number of years now! Welcome to the conversation.