Most Kubernetes problems are not Kubernetes problems
By the time a team calls in help because “Kubernetes is unreliable,” the platform itself is rarely the root cause. The failures usually trace back to a handful of patterns that were skipped early because they did not block the first successful deployment. They just make every deployment after that one riskier.
GitOps over push based deploys
A pipeline that runs kubectl apply or helm upgrade directly from CI gives you a deployment, but no reliable record of what is actually running in the cluster versus what the pipeline last pushed. A GitOps approach, where a controller such as Argo CD or Flux continuously reconciles the cluster state against a Git repository, means the repository is always the source of truth. Drift gets detected and corrected automatically instead of discovered during an incident.
Environment parity that actually holds
Staging environments that quietly diverge from production, different resource limits, different node pools, different versions of a dependent service, are one of the most common reasons a deployment that passed every test still breaks in production. Parity does not mean identical scale. It means the configuration, not just the code, is templated and promoted the same way through every environment, so the only thing that changes between staging and production is the values file.
A rollback plan that does not depend on someone remembering the steps
Canary or blue green deployment strategies matter less than most teams think if there is no automated health check deciding when to roll back. The pattern that actually reduces incident time is tying rollback to an automated signal, an error rate, latency, or a readiness probe failing past a threshold, rather than a person watching a dashboard and making the call under pressure.
Secrets management that survives an audit
Secrets stored as plain Kubernetes secrets, checked into a values file, or passed as environment variables in a CI pipeline are a common finding in almost every platform review. A secrets manager such as Azure Key Vault or AWS Secrets Manager, integrated through a CSI driver or an operator that syncs into the cluster, closes this gap without adding much operational overhead once it is set up.
Resource requests, limits, and the autoscaler that gets it wrong
Horizontal pod autoscaling only works as well as the resource requests it is scaling against. Pods with no requests set, or requests copied from another service without thought, produce autoscaling behavior that looks broken but is actually working exactly as configured. Getting this right takes real load data, not a guess, which is why it is worth revisiting a few weeks after go live rather than only at launch.
A short example
On a platform I worked on for a mid size enterprise, deployments were technically automated but still triggered a manual Slack message asking the team to keep an eye on it for twenty minutes after every release. The pipeline was not missing automation, it was missing a defined, automated rollback condition. Adding a simple error rate threshold that triggered an automatic rollback removed the twenty minute manual watch entirely and cut the average time to recover from a bad deploy from around fifteen minutes to under two.
Getting the platform right is only half the job. The other half is the delivery structure around it, covered in this post on IT project delivery. And if this platform work is part of a larger cloud migration, the budget realities are covered in what a migration actually costs.
If you are working through a platform reliability problem and want a second set of eyes, reach out through the contact page.
Leave a comment