Running Kubernetes in Production: Lessons Learned

The Kubernetes Production Reality

Kubernetes has become the de facto standard for container orchestration, but running it in production comes with unique challenges. After deploying dozens of production Kubernetes clusters, we've learned valuable lessons about reliability, security, and operations.

Cluster Architecture Decisions

Managed vs Self-Hosted

For most organizations, managed Kubernetes services (EKS, GKE, AKS) are the pragmatic choice. They handle control plane operations, updates, and availability. Self-host only if you have specific requirements and dedicated platform engineering team.

Multi-Tenancy Strategy

Decide between single large cluster with namespaces or multiple smaller clusters. Multiple clusters provide stronger isolation but increase operational overhead. Use namespaces with RBAC and network policies for soft multi-tenancy.

Essential Production Practices

• Always set resource requests and limits for CPU and memory
• Implement pod disruption budgets for high availability
• Use horizontal pod autoscaling based on metrics
• Implement health checks (liveness, readiness, startup probes)
• Use init containers for setup tasks
• Enable pod security policies or admission controllers

Networking Considerations

CNI Plugin Selection

Choose appropriate Container Network Interface plugin. Calico for network policies, Cilium for advanced networking and observability, or cloud provider's native CNI for simplicity. Test thoroughly before production.

Ingress and Service Mesh

Use ingress controllers (NGINX, Traefik, AWS ALB) for external traffic. Consider service mesh (Istio, Linkerd) for microservices with complex communication patterns, though they add operational complexity.

Storage Management

Understand persistent volume lifecycle. Use storage classes for dynamic provisioning. Back up persistent volumes regularly. For stateful applications like databases, consider running them outside Kubernetes unless you have strong Kubernetes operators.

Observability Stack

Metrics

Deploy Prometheus for metrics collection, Grafana for visualization. Monitor both cluster-level metrics (node CPU, memory, disk) and application metrics. Set up alerting for critical conditions.

Logging

Centralize logs with ELK stack or managed solutions like CloudWatch Logs. Structure logs as JSON for better querying. Implement log retention policies to control costs.

Tracing

Use distributed tracing (Jaeger, Tempo) to understand request flows across microservices. Essential for debugging performance issues in complex systems.

Security Hardening

• Enable RBAC and follow principle of least privilege
• Use network policies to restrict pod-to-pod communication
• Scan container images for vulnerabilities regularly
• Rotate secrets and use external secret managers (Vault, AWS Secrets Manager)
• Enable audit logging for compliance
• Restrict privileged containers

Disaster Recovery

Backup etcd regularly. Document and test cluster recovery procedures. Use infrastructure as code (Terraform, Pulumi) for reproducible cluster creation. Implement multi-region strategies for critical applications.

Cost Optimization

Right-size node pools and pod resources. Use spot instances for fault-tolerant workloads. Implement cluster autoscaling. Monitor and optimize network egress costs. Use pod priority and preemption for cost-effective scheduling.

Common Pitfalls to Avoid

• Running too many small pods (overhead issues)
• Not setting resource limits (noisy neighbor problems)
• Ignoring pod disruption budgets (availability issues during updates)
• Over-complicating with service mesh too early
• Insufficient monitoring and alerting

Conclusion

Kubernetes is powerful but complex. Start simple, add capabilities as needed, invest in observability early, and document everything. With proper planning and operations, Kubernetes can provide reliable, scalable infrastructure for your applications.