Kubernetes is fast becoming an industry standard, with up to 94% of organizations deploying their services and applications on the container orchestration platform, per a survey. One of the key reasons companies deploy on Kubernetes is standardization, which lets advanced users double productivity gains.
Standardizing on Kubernetes gives organizations the ability to deploy any workload, anywhere. But there was a missing piece: The technology assumed that workloads were ephemeral, meaning that only stateless workloads could be safely deployed on Kubernetes. However, the community recently changed the paradigm and brought features such as StatefulSets and Storage Classes, which make using data on Kubernetes possible.
While running stateful workloads on Kubernetes is possible, it is still challenging. In this article, I provide ways to make it happen and why it is worth it.
Do it progressively
Kubernetes is on its way to being as popular as Linux and the de facto way of running any application, anywhere, in a distributed fashion. Using Kubernetes involves learning a lot of technical concepts and vocabulary. For instance, newcomers might struggle with the many Kubernetes logical units such as containers, pods, nodes and clusters.
If you are not running Kubernetes in production yet, don’t jump directly into data workloads. Instead, start with moving stateless applications to avoid losing data when things go sideways.
If you can’t find an operator that matches your needs, don’t worry, because most of them are open source.
Understand the limitations and specificities
Once you are familiar with general Kubernetes concepts, dive into the specifics for stateful concepts. For example, because applications may have different storage needs, such as performance or capacity requirements, you must provide the correct underlying storage system.
What the industry generally calls storage “profiles” is termed Storage Classes in Kubernetes. They provide a way to describe the different types of classes a Kubernetes cluster can access. Storage classes can have different quality-of-service levels, such as I/O operations per second per GiB, backup policies or arbitrary policies such as binding modes and allowed topologies.
Another critical component to understand is StatefulSet. It is the Kubernetes API object used to manage stateful applications and offers key features such as:
- Stable, unique network identifiers that let you keep track of volume, and allows you to detach and reattach them as you please.
- Stable, persistent storage so that your data is safe.
- Ordered, graceful deployment and scaling, which is required for many Day 2 operations.
While StatefulSet has been a successful replacement for the infamous PetSet (now deprecated), it is still imperfect and has limitations. For example, the StatefulSet controller has no built-in support for volume (PVC) resizing — which is a major challenge if the size of your application dataset is about to grow above the current allocated storage capacity. There are workarounds, but such limitations must be understood well ahead of time so that the engineering team knows how to handle them.
Come up with a plan
Once you are comfortable with Kubernetes stateful concepts, you can progressively migrate your data workloads in a specific order. This allows you to learn from your mistakes and avoid being overwhelmed, because not all data technologies are equally easy to run on Kubernetes.
Established technologies, such as databases and storage, should be migrated first, and emerging tech, such as AI and ML, should be done last. This is reflected in a recent report, which found database and persistent storage are the two most-run data workloads on Kubernetes. The main reason is the lack of tooling for Day 2 operations. We will explore this in the next section.
Check for operator availability
Moving your stateful workloads to Kubernetes is only half the job — also known as Day 1. Now you need to handle Day 2 operations (one of the most discussed topics at the last KubeCon). This is where things get tricky. There are tons of Day 2 operations that Kubernetes cannot handle natively such as patching and upgrading, backup and recovery, log processing, monitoring, scaling and tuning.
All these operations are application specific. For example, a PostgreSQL and MySQL cluster will require two completely different approaches when picking a new primary server in an HA cluster configuration. Kubernetes cannot possibly know all the application’s specific Day 2 operations. This is where operators come in.
Operators are programmable extensions that perform operations that Kubernetes cannot handle natively. Operators provide intelligent, dynamic management capabilities by extending the functionality of the Kubernetes API. One of the most common uses is conducting these Day 2 operations. These operators aren’t developed by the Kubernetes maintainers but by third-party developers and organizations.
Before moving a data workload to Kubernetes, make sure there is an operator for it. OperatorHub does a great job of indexing them. With 282 operators available on the site, the distribution echoes what we discussed earlier: Some workloads have supporting tools, and some don’t. For example, the database category has 38 operators — there are eight for PostgreSQL alone — while the entire ML/AI category only has seven.
Pick the right level of operator capability
Having an operator for your technology isn’t enough, because they can have different capabilities and often exist at various levels of maturity. The OperatorFramework suggests a capability model that categorizes operators according to their features:
- Level 1: Works for basic installation, such as automated application provisioning and configuration management.
- Level 2: Supports seamless upgrades, patches and minor version upgrades.
- Level 3: Handles the full app and storage lifecycle (backup, failure recovery, etc.).
- Level 4: Provides deep insights, metrics, alerts, log processing and workload analysis.
- Level 5: Offers automatic horizontal/vertical scaling, auto-config tuning, abnormality detection and scheduling tuning.
When choosing an operator, make sure its capabilities match your needs. If you are unsure which level is right for you, the Data on Kubernetes Report 2022 found that most organizations are looking for operators that are at least at Level 3. Having a backup for your stateful workloads sounds like a good idea.
If you can’t find an operator that matches your needs, don’t worry because most of them are open source. You can extend existing operators’ capabilities with internal development or, even better, contribute to the open source project.
Understand the operator
Operators’ extensibility is their strength, but it’s also their weakness. The lack of standards means they are programmed differently, so you must look at their config files to pick the format you like best.
What’s more, operators may use different technical routes to achieve the same goal. For example, one of the eight PostgreSQL operators, CloudNativePG, does not use StatefulSets, and instead uses its own custom controller. That’s quite unexpected considering that StatefulSets is the foundation for stateful workloads on Kubernetes.
Its developers decided to go with this design because of the inability of StatefulSet to resize PVCs (as we discussed earlier). As the operator documentation explains, picking “different [design directions] lead to other compromises.” So when picking an operator, be sure to understand its implementation and trade-offs, and go with the one you are the most comfortable with.
It’s worth the effort
As you can see, running data on Kubernetes isn’t always easy, but the good news is that it’s worth the hard work: 54% of surveyed organizations attributed more than 10% of their revenue to the fact that they run data on Kubernetes. What’s more, 33% said it has a transformative impact on productivity and another 51% saw a significant positive impact.
As organizations increasingly adopt multicloud infrastructure to optimize their cost and infrastructure performance, Kubernetes has become the tool of choice. With an estimated 66% of countries having some sort of data privacy and consumer rights legislation, which often requires enforcing data sovereignty, companies must increasingly host user data in the countries they operate in. Kubernetes is here to stay.