What is Kubernetes? Container Orchestration at Scale
Kubernetes runs your Docker containers across a cluster of machines, handles scheduling, health checking, rolling deployments, auto-scaling, and self-healing. When your application outgrows a single server, Kubernetes is how it runs on ten.
Docker solved the "it works on my machine" problem by packaging applications and their environments into containers. That works well until you have more containers than one machine can run, or you need multiple instances of a service for redundancy and load distribution, or a container crashes and needs to be restarted somewhere, or traffic spikes and you need to scale out quickly.
Running Docker containers on one machine is manageable. Running them on ten machines, keeping them healthy, distributing traffic between them, deploying new versions without downtime, and scaling individual services in response to load — that requires a different tool.
Kubernetes is that tool. It is a container orchestration platform that manages how containers are scheduled and run across a cluster of machines. You tell Kubernetes what you want: run four instances of my API container, ensure at least two are always available, make this accessible on port 443, use this secret for the database password. Kubernetes figures out which machines have capacity, places the containers, monitors their health, restarts them when they fail, and adjusts when you change what you want.
Google built Kubernetes based on their internal system called Borg, which they had used to run their infrastructure for a decade before open-sourcing it in 2014. The CNCF (Cloud Native Computing Foundation) now maintains it. Every major cloud provider runs managed Kubernetes: AWS EKS, Google GKE, Azure AKS. Kubernetes has become the standard substrate for production container deployments.
The Core Concepts
Cluster, Nodes, and Pods
A cluster is a set of machines (nodes) running Kubernetes. Every cluster has a control plane (the Kubernetes brain: scheduler, API server, etcd database, controllers) and worker nodes (the machines that actually run your containers).
A Pod is the smallest deployable unit in Kubernetes. A Pod runs one or more containers that share a network namespace and storage volumes. Most Pods contain one container. Pods that do run multiple containers are usually a main application container and a sidecar (a helper container for logging, monitoring, or proxying).
Pods are ephemeral. Kubernetes can kill them and start new ones on different nodes at any time. When a node fails, Kubernetes reschedules the Pods that were running on it to healthy nodes. You do not configure individual Pods directly — you configure higher-level objects that manage Pods.
Deployments
A Deployment describes the desired state for a set of Pods. How many replicas. Which container image. What resource limits. What update strategy. Kubernetes continuously reconciles the actual state toward the desired state. If you say "I want 4 replicas of my API" and a node fails, killing 2 of them, Kubernetes immediately schedules 2 new Pods on healthy nodes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: production
labels:
app: api
version: "1.2.0"
spec:
replicas: 4
selector:
matchLabels:
app: api
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: api
version: "1.2.0"
spec:
containers:
- name: api
image: ghcr.io/company/api:1.2.0
ports:
- containerPort: 3000
env:
- name: NODE_ENV
value: production
- name: PORT
value: "3000"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-url
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: redis-url
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "512Mi"
readinessProbe:
httpGet:
path: /health/ready
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health/live
port: 3000
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
replicas: 4 runs four identical Pod copies. maxSurge: 1 allows one additional Pod during rolling updates (briefly running 5). maxUnavailable: 0 means no Pods go offline during updates — every old Pod waits for a new one to pass readiness checks before terminating. Zero-downtime rolling deployment.
resources.requests tells the scheduler how much CPU and memory this container needs. The scheduler uses requests to find a node with sufficient capacity. resources.limits is the cap the container cannot exceed. A container exceeding its memory limit is OOMKilled (out of memory killed). Setting requests and limits correctly is one of the most important operational tasks in Kubernetes.
readinessProbe determines whether the Pod should receive traffic. Kubernetes does not add the Pod to the load balancer until the readiness probe passes. If a readiness probe starts failing on a running Pod, Kubernetes removes it from the load balancer without killing it. livenessProbe determines whether the Pod should be restarted. Failing a liveness probe kills and replaces the Pod.
The preStop hook and terminationGracePeriodSeconds handle graceful shutdown. When Kubernetes needs to terminate a Pod, it sends SIGTERM and waits terminationGracePeriodSeconds for the process to exit cleanly. The preStop sleep of 5 seconds gives the load balancer time to stop routing new traffic to the Pod before the application starts shutting down.
Services
A Service provides a stable network endpoint for Pods. Pods have ephemeral IP addresses — they change when Pods are killed and rescheduled. Services get a stable IP and DNS name that routes to whatever Pods match their selector.
apiVersion: v1
kind: Service
metadata:
name: api
namespace: production
spec:
selector:
app: api
ports:
- port: 80
targetPort: 3000
protocol: TCP
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: api-external
namespace: production
spec:
selector:
app: api
ports:
- port: 443
targetPort: 3000
type: LoadBalancer
ClusterIP is the default — the service is reachable only inside the cluster. Other services reference api.production.svc.cluster.local to reach it. LoadBalancer provisions a cloud load balancer (AWS ELB, GCP Load Balancer) with an external IP. NodePort exposes the service on a high port on each node (less commonly used directly).
Ingress
An Ingress defines HTTP/HTTPS routing rules from external traffic to services. Instead of a LoadBalancer per service, one Ingress controller (NGINX, Traefik, AWS ALB) handles all routing.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
namespace: production
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rate-limit: "100"
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: api-tls
rules:
- host: api.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: frontend
port:
number: 80
cert-manager.io/cluster-issuer: "letsencrypt-prod" automatically provisions and renews TLS certificates from Let's Encrypt. One annotation on the Ingress object. No manual certificate management.
ConfigMaps and Secrets
apiVersion: v1
kind: ConfigMap
metadata:
name: api-config
namespace: production
data:
NODE_ENV: "production"
PORT: "3000"
LOG_LEVEL: "info"
CORS_ORIGINS: "https://app.example.com"
---
apiVersion: v1
kind: Secret
metadata:
name: api-secrets
namespace: production
type: Opaque
stringData:
database-url: "postgresql://user:password@postgres:5432/myapp"
redis-url: "redis://:password@redis:6379/0"
jwt-access-secret: "your-32-character-minimum-secret-key"
jwt-refresh-secret: "different-32-character-refresh-secret"
ConfigMaps store non-sensitive configuration. Secrets store sensitive data. Kubernetes base64-encodes Secret values (this is not encryption). For real secret management, use external providers: AWS Secrets Manager with the External Secrets Operator, HashiCorp Vault, or Sealed Secrets. These systems store encrypted secrets outside the cluster and inject them at runtime.
Apply YAML to the cluster:
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml
kubectl apply -f -k ./overlays/production
kubectl apply -k uses Kustomize, which layers environment-specific patches over a base configuration. One base Deployment definition. Production overlay changes the image tag and replica count. Staging overlay uses different resource limits. No duplicated YAML.
The kubectl Workflow
kubectl get pods -n production
kubectl get pods -n production -w
kubectl get deployments -n production
kubectl get services -n production
kubectl get ingress -n production
kubectl get all -n production
kubectl describe pod api-7d9c8f6b4-xk2p9 -n production
kubectl logs api-7d9c8f6b4-xk2p9 -n production
kubectl logs api-7d9c8f6b4-xk2p9 -n production --previous
kubectl logs -l app=api -n production --tail=100
kubectl exec -it api-7d9c8f6b4-xk2p9 -n production -- sh
kubectl get events -n production --sort-by='.lastTimestamp'
kubectl top pods -n production
kubectl top nodes
-w (watch) streams updates. kubectl get pods -n production -w shows Pod status changes in real time as a deployment rolls out. When debugging a failing deployment, this is the first window to open.
kubectl describe pod <name> shows the full Pod specification, conditions, and importantly the Events section — Kubernetes events that explain what happened: image pull failures, OOMKill events, scheduling failures, probe failures.
kubectl logs --previous shows logs from the previous container instance — critical when a container crashes on startup before you can capture live logs.
Namespaces and Multi-Tenancy
Namespaces partition a cluster into isolated environments. Resources in different namespaces do not conflict with each other.
kubectl create namespace production
kubectl create namespace staging
kubectl create namespace monitoring
kubectl get pods --all-namespaces
kubectl config set-context --current --namespace=production
kubectl get resourcequota -n production
kubectl get limitrange -n production
apiVersion: v1
kind: ResourceQuota
metadata:
name: production-quota
namespace: production
spec:
hard:
requests.cpu: "8"
requests.memory: "16Gi"
limits.cpu: "16"
limits.memory: "32Gi"
pods: "50"
services: "20"
ResourceQuotas limit total resource consumption per namespace. This prevents one team's runaway workload from consuming all cluster resources and starving other teams. LimitRanges set default requests and limits for containers that do not specify their own.
Horizontal Pod Autoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 70
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 4
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 60
The HPA watches CPU and memory utilization. When average CPU across all Pods exceeds 60%, it adds Pods. When it drops below 60%, it removes Pods. The stabilizationWindowSeconds for scale-down prevents thrashing: it waits 5 minutes of sustained low utilization before removing Pods. Scale-up is more aggressive — 4 new Pods per minute is a fast response to traffic spikes.
Persistent Storage
Stateful workloads (databases, message queues, file storage) need storage that persists beyond Pod restarts. PersistentVolumes and PersistentVolumeClaims provide this.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-pvc
namespace: production
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 50Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: production
spec:
serviceName: postgres
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16-alpine
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secrets
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2"
memory: "4Gi"
volumeClaimTemplates:
- metadata:
name: postgres-storage
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3
resources:
requests:
storage: 50Gi
StatefulSets manage stateful applications. Unlike Deployments, StatefulSets give Pods stable identities (postgres-0, postgres-1) and stable storage. Each Pod gets its own PersistentVolumeClaim that is not deleted when the Pod is rescheduled. volumeClaimTemplates auto-creates PVCs for each replica. For databases in Kubernetes, StatefulSets are the right primitive.
ReadWriteOnce means one node at a time can mount the volume for reading and writing. This is appropriate for a single-instance database. For shared storage across multiple Pods simultaneously, you need ReadWriteMany storage (NFS, AWS EFS, GlusterFS).
Deploying with Helm
Helm is the package manager for Kubernetes. Instead of managing raw YAML files, you install pre-packaged applications (charts) with a single command. Community charts exist for PostgreSQL, Redis, NGINX, Prometheus, Grafana, cert-manager, and hundreds of other common infrastructure components.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install nginx-ingress ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
helm install postgres bitnami/postgresql \
--namespace production \
--set auth.postgresPassword=secretpassword \
--set primary.persistence.size=50Gi \
--set primary.resources.requests.memory=1Gi
helm install redis bitnami/redis \
--namespace production \
--set auth.password=redispassword \
--set master.persistence.size=10Gi
helm list -A
helm upgrade postgres bitnami/postgresql --namespace production --reuse-values --set primary.persistence.size=100Gi
helm rollback postgres 1 --namespace production
helm uninstall postgres --namespace production
For your own application, create a Helm chart:
helm create my-api
This generates a directory with Chart.yaml (metadata), values.yaml (defaults), and templates/ (Kubernetes manifests with Go template syntax). Customize values.yaml for your application. Create environment-specific overrides in separate values files.
Monitoring Kubernetes
The standard monitoring stack: Prometheus for metrics collection, Grafana for dashboards, and AlertManager for alerting.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set grafana.adminPassword=yoursecurepassword
kube-prometheus-stack installs Prometheus, Grafana, AlertManager, and a set of pre-configured dashboards and alerts for Kubernetes cluster health in one command. You get CPU, memory, network, and Pod status dashboards immediately.
Your application exposes metrics in Prometheus format at /metrics. A ServiceMonitor tells Prometheus to scrape it:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api
namespace: production
spec:
selector:
matchLabels:
app: api
endpoints:
- port: http
path: /metrics
interval: 15s
Kubernetes is the most complex piece of infrastructure most developers interact with. The learning curve is steep. The operational complexity is real. But for applications that run across multiple servers and need health management, auto-scaling, rolling deployments, and declarative configuration, there is no viable alternative at this level of capability and ecosystem maturity. Managed Kubernetes from cloud providers removes the control plane management burden, leaving you with cluster configuration and application deployment — both learnable skills with significant returns.
Jobs, CronJobs, and Batch Workloads
Not all workloads are long-running servers. Kubernetes Jobs run a task to completion. CronJobs run Jobs on a schedule.
apiVersion: batch/v1
kind: Job
metadata:
name: db-migration
namespace: production
spec:
backoffLimit: 3
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: OnFailure
containers:
- name: migrate
image: ghcr.io/company/api:1.2.0
command: ["node", "dist/database/migrate.js"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-url
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: daily-digest
namespace: production
spec:
schedule: "0 8 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: digest
image: ghcr.io/company/api:1.2.0
command: ["node", "dist/jobs/daily-digest.js"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-url
Jobs use restartPolicy: OnFailure or restartPolicy: Never. Deployments use Always. A Pod in a Job that exits with code 0 is marked as succeeded and not restarted. Exiting with nonzero triggers a restart up to backoffLimit times. activeDeadlineSeconds is a hard timeout — the Job fails if it runs longer than this.
concurrencyPolicy: Forbid prevents a CronJob from starting a new run if a previous run is still running. For jobs that must not run concurrently (database batch jobs, report generation), this prevents overlap.
Rolling Deployments and Rollbacks
kubectl set image deployment/api api=ghcr.io/company/api:1.3.0 -n production
kubectl rollout status deployment/api -n production
kubectl rollout history deployment/api -n production
kubectl rollout history deployment/api -n production --revision=3
kubectl rollout undo deployment/api -n production
kubectl rollout undo deployment/api --to-revision=2 -n production
kubectl rollout pause deployment/api -n production
kubectl rollout resume deployment/api -n production
kubectl rollout status blocks and streams progress. It exits 0 when the rollout completes and 1 if it times out. Use it in CI/CD pipelines to wait for a deployment before proceeding to smoke tests.
rollout pause freezes an in-progress rolling update. Useful if you notice something wrong partway through — you can pause, diagnose, and either resume or roll back.
Kubernetes stores rollout history. rollout undo with no revision rolls back to the previous version. --to-revision specifies an exact revision. The number of stored revisions is controlled by spec.revisionHistoryLimit in the Deployment (default 10).
RBAC: Role-Based Access Control
Kubernetes RBAC controls who can perform which operations on which resources.
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-sa
namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: api-role
namespace: production
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["api-secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: api-role-binding
namespace: production
subjects:
- kind: ServiceAccount
name: api-sa
namespace: production
roleRef:
kind: Role
name: api-role
apiGroup: rbac.authorization.k8s.io
ServiceAccounts are identities for Pods. A Pod runs as its ServiceAccount. The Role grants specific permissions within a namespace. The RoleBinding links the Role to the ServiceAccount. Your API Pod can read ConfigMaps and the specific api-secrets Secret, but cannot create, modify, or delete any Kubernetes objects. The principle of least privilege applied at the platform level.
ClusterRoles and ClusterRoleBindings apply across all namespaces. Use them for cluster-wide resources (nodes, persistent volumes) or operators that need access across namespaces.
Network Policies
By default, all Pods in a cluster can communicate with all other Pods. NetworkPolicy restricts this.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-network-policy
namespace: production
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- protocol: TCP
port: 3000
egress:
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
- to: []
ports:
- protocol: TCP
port: 443
- protocol: TCP
port: 53
- protocol: UDP
port: 53
This NetworkPolicy on the api Pods allows: inbound traffic only from the nginx ingress controller on port 3000, outbound traffic to postgres on 5432 and redis on 6379, outbound HTTPS (443) and DNS (53) for external API calls. Everything else is denied. If a container inside an API Pod is compromised, it cannot reach other services in the cluster beyond what the application legitimately needs.
NetworkPolicy requires a CNI plugin that implements it. Calico, Cilium, and Weave Net all support NetworkPolicy. The default Flannel CNI does not. On managed Kubernetes (EKS, GKE, AKE), network policy support is generally available with the right configuration.
Kubernetes in Production: The Reality
Kubernetes adds real operational complexity. The learning curve is steep. A team that does not already understand Linux networking, container concepts, and distributed systems principles will struggle to operate a production Kubernetes cluster reliably.
For small teams or applications running on one or two servers, Docker Compose on a single server is genuinely the right choice. The complexity of Kubernetes is only justified when you have: multiple services that need independent scaling, multi-region or multi-zone deployments, teams large enough that self-service deployments matter, or traffic patterns that require auto-scaling.
Managed Kubernetes (EKS, GKE, AKE) removes control plane management. You do not run etcd or the API server. You configure node pools, namespaces, and workloads. The cloud provider handles availability of the control plane. This is the right starting point for teams adopting Kubernetes — the operational overhead is significantly lower than self-managed.
Platform-as-a-service options like Railway, Render, and Fly.io provide Kubernetes-like capabilities (container deployment, health checking, auto-scaling, zero-downtime deploys) without requiring you to understand Kubernetes directly. For applications that fit their constraints, these platforms are faster to operate than raw Kubernetes.
Kubernetes is infrastructure that makes other things possible. It does not make your application better on its own. The value is in the operational capabilities it provides for large, complex deployments — and the ecosystem of tooling built on top of it.
Kubernetes Networking: Services, DNS, and Ingress Deep Dive
Every Pod gets an IP address assigned by the cluster's CNI plugin. That IP is ephemeral — when the Pod dies and a new one starts, it gets a different IP. Services provide stable endpoints.
Kubernetes runs a DNS server (CoreDNS) in every cluster. When you create a Service called api in the production namespace, DNS automatically resolves:
api— within the same namespaceapi.production— short form from another namespaceapi.production.svc.cluster.local— fully qualified name from anywhere in the cluster
Your application code connecting to the database can hardcode postgres.production.svc.cluster.local:5432 and it will work regardless of which node the Postgres Pod runs on, when it restarts, or what IP it gets. This DNS-based service discovery is fundamental to how microservices communicate in Kubernetes.
kubectl run debug --image=busybox --rm -it --restart=Never -- sh
nslookup api.production.svc.cluster.local
wget -qO- http://api.production.svc.cluster.local/health/live
kubectl get endpoints api -n production
kubectl describe service api -n production
kubectl get endpoints shows the actual Pod IPs that the Service is routing to. When a Pod fails its readiness probe, its IP disappears from the endpoints list and traffic stops flowing to it — all automatically, before your users notice anything.
Resource Management at Scale
Kubernetes scheduling decisions depend on accurate resource specifications. Getting them wrong causes two classes of problems: over-specifying resources means fewer Pods fit per node (expensive), under-specifying means Pods get OOMKilled or throttled.
kubectl top pods -n production
kubectl top pods -n production --sort-by=cpu
kubectl top pods -n production --sort-by=memory
kubectl describe node worker-node-1 | grep -A5 "Allocated resources"
kubectl describe pod api-7d9c8f6b4-xk2p9 -n production | grep -A10 "Requests"
The output of kubectl describe node shows how much CPU and memory is allocated to running Pods versus the node's total capacity. When a node is at 95% allocation, new Pods that need resources cannot be scheduled there. The scheduler moves to another node or, if no node has capacity, the Pod stays in Pending state.
LimitRange sets default values for Pods that do not specify their own:
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: production
spec:
limits:
- type: Container
default:
cpu: "200m"
memory: "256Mi"
defaultRequest:
cpu: "50m"
memory: "64Mi"
max:
cpu: "2"
memory: "4Gi"
min:
cpu: "10m"
memory: "32Mi"
Any container deployed without explicit resource specifications inherits these defaults. The max constraint prevents rogue workloads from requesting unlimited resources. Without LimitRange, a developer who forgets resource limits in their Deployment YAML can accidentally request all CPU on a node.
Kubernetes Operators
Operators extend Kubernetes with custom resources and controllers that manage stateful applications. Instead of manually running database migrations, promoting replicas, and handling failover, an Operator automates it.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: production
spec:
instances: 3
storage:
size: 50Gi
storageClass: gp3
postgresql:
parameters:
max_connections: "200"
shared_buffers: "256MB"
effective_cache_size: "768MB"
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: s3://my-backups/postgres
s3Credentials:
accessKeyId:
name: s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-creds
key: SECRET_ACCESS_KEY
The CloudNativePG Operator watches for Cluster resources and manages a PostgreSQL primary with two synchronous replicas, automated backups to S3, point-in-time recovery, and automatic failover. You describe what you want in a YAML file. The Operator handles the operational complexity.
Operators exist for PostgreSQL (CloudNativePG), Redis (Redis Operator), Kafka (Strimzi), Elasticsearch (Elastic Cloud on Kubernetes), Prometheus, cert-manager, and dozens of other common infrastructure components. Installing an Operator with Helm and then declaring resources through custom YAML is significantly simpler than writing StatefulSets and backup scripts manually.
GitOps with Flux or ArgoCD
GitOps is the practice of using a Git repository as the single source of truth for cluster state. Every change to what runs in the cluster goes through a pull request. An automated controller watches the repository and reconciles the cluster to match.
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace
kubectl port-forward svc/argocd-server -n argocd 8080:443
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-api
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/company/k8s-manifests
targetRevision: HEAD
path: apps/production/api
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
ArgoCD watches the apps/production/api directory in your Git repository. When a file changes (a new image tag in a Deployment, an updated ConfigMap, a new Service), ArgoCD applies the change to the cluster automatically. selfHeal: true means if someone manually modifies a resource in the cluster (bypassing Git), ArgoCD reverts it. The repository is always truth.
This model makes deployments auditable, rollbacks trivial (git revert the image tag change), and cluster state reproducible (destroy the cluster and recreate it from the repository).
Multi-Cluster and Federation
Large organizations run multiple Kubernetes clusters — one per region, one per environment, or one per team. Managing multiple clusters requires tools beyond single-cluster kubectl.
kubectx and kubens are the practical first step — simple tools that make switching between clusters and namespaces fast:
kubectx list
kubectx production-us-east
kubectx staging-eu-west
kubens production
kubens monitoring
Cluster API (CAPI) manages the lifecycle of Kubernetes clusters themselves as Kubernetes resources. You create a Cluster resource and CAPI provisions the VMs, installs Kubernetes, and manages upgrades. The cluster is described declaratively and reconciled like any other Kubernetes object.
For most teams, multi-cluster Kubernetes is years away. Start with a single managed cluster. Learn the operational patterns. Understand what actually breaks in production. Multi-cluster adds significant complexity and is only justified when a single cluster's limits are genuinely reached.
Kubernetes in the Real World: Common Failure Modes
Knowing what breaks in production Kubernetes clusters saves hours of debugging.
Pods stuck in Pending. The scheduler cannot find a node that satisfies the Pod's requirements. Common causes: resource requests exceed what any node has available, a node selector or affinity rule matches no nodes, a PVC cannot be bound because no storage matches the storage class. kubectl describe pod <name> shows the scheduler's reason in the Events section.
OOMKilled containers. The container exceeded its memory limit. Kubernetes kills it and restarts it. kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' returns OOMKilled. Either increase the memory limit, or find the memory leak causing the container to grow.
CrashLoopBackOff. The container starts, crashes, starts, crashes. Kubernetes backs off the restart interval exponentially. kubectl logs <pod> --previous shows logs from the crashed container, which usually reveals the error. Common causes: missing environment variables, database connection failures at startup, application bug in initialization code.
ImagePullBackOff. Kubernetes cannot pull the container image. Either the image does not exist at that tag, the registry requires authentication that is not configured, or there is a network policy blocking outbound traffic to the registry. kubectl describe pod <name> shows the specific error from the image pull attempt.
Service not routing traffic. The selector does not match any Pod labels. kubectl get endpoints <service-name> shows <none> if no Pods match. Compare the Service's spec.selector with the Pod's metadata.labels exactly — a label mismatch is invisible from the Service's perspective.
Resource version conflicts on apply. When two processes apply to the same resource simultaneously, the second apply gets a conflict error. Use server-side apply (kubectl apply --server-side) which handles merging correctly and avoids conflicts in GitOps scenarios.
kubectl get events -n production --sort-by='.lastTimestamp' | tail -20
kubectl get pods -n production -o wide
kubectl describe pod <crashing-pod> -n production
kubectl logs <crashing-pod> -n production --previous --tail=100
kubectl get endpoints -n production
kubectl auth can-i create pods --namespace production --as system:serviceaccount:production:api-sa
kubectl auth can-i checks whether a specific service account has permission to perform an action. When a Pod is failing because of RBAC, this command tells you exactly what permission is missing.
Cost Management
Kubernetes clusters on cloud providers get expensive. Reserved instances and committed use discounts help, but so does right-sizing your workloads.
VPA (Vertical Pod Autoscaler) analyzes historical resource usage and recommends (or automatically adjusts) CPU and memory requests to match actual usage. Running VPA in recommendation mode for a week shows you where resources are over-provisioned without making any changes.
kubectl describe vpa api-vpa -n production
Karpenter (AWS) and Cluster Autoscaler are node autoscalers — they add and remove nodes based on pending workloads and node utilization. Cluster Autoscaler removes nodes where all Pods can be scheduled elsewhere, reducing the node count (and cost) during low-traffic periods.
Kubecost or OpenCost add cost visibility to Kubernetes — how much each namespace, deployment, and even Pod costs per month based on actual resource consumption multiplied by cloud instance pricing. Without cost visibility, Kubernetes clusters tend to accumulate over-provisioned workloads that nobody audits.
The Kubernetes ecosystem is enormous. The surface area is deliberately large because it is designed to be a platform — the foundation on top of which other tools build. Learning the core concepts (Pods, Deployments, Services, ConfigMaps, Secrets, Ingress) gets you 80% of the way. The other 20% is the operational experience that comes from running it in production and dealing with the failure modes described above.
Kubernetes Security Hardening
A default Kubernetes cluster is not secure. Hardening requires deliberate configuration at multiple layers.
Pod Security Standards replace the deprecated Pod Security Policy and restrict what Pods can do:
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
The restricted policy rejects Pods that run as root, that mount sensitive host paths, or that use host networking or host ports. Any Deployment that does not comply is rejected at admission. This forces every team deploying to the production namespace to use non-root containers with appropriate security contexts.
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true mounts the container's root filesystem as read-only. If a process inside the container tries to write to the filesystem (not to a mounted volume), it gets a permission error. This limits what a compromised container can do — it cannot write malware to the filesystem, cannot modify configuration files, and cannot accumulate state between requests.
capabilities.drop: ALL removes all Linux capabilities from the process. A process with no capabilities cannot bind to ports below 1024, cannot change file ownership, cannot load kernel modules. Combined with running as a non-root user, this is defense in depth against container escape.
Secret encryption at rest encrypts the etcd database contents so that secrets are not stored in plaintext:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
Without encryption at rest, anyone with access to the etcd snapshot can read all your Kubernetes secrets in plaintext. Cloud providers (EKS, GKE, AKE) enable encryption at rest by default. Self-managed clusters require explicit configuration.
Kubernetes security is a deep field. The CNCF's security whitepaper, CIS Kubernetes Benchmark, and NSA/CISA Kubernetes Hardening Guidance are the authoritative references for production security configuration.
The learning investment pays off when those capabilities are what your project actually needs.