resources:
  requests:
    cpu: 500m
    memory: 256Mi

The scheduler uses requests to determine node placement. A request of 500m means the container needs half a CPU core. The scheduler won't place the container on a node without 500m available.

Resource Limits

Limits define the maximum resources a container can consume:

resources:
  limits:
    cpu: 1000m
    memory: 512Mi

The container runtime enforces limits. Exceeding them causes CPU throttling or out-of-memory termination.

Why This Matters

Without requests, the scheduler overbooks nodes. Too many containers end up on one node, each starving for CPU and memory. Performance degrades. Requests prevent this.

Without limits, a misbehaving container consumes all node resources, starving other containers. Limits prevent runaway consumption.

Getting Requests Right

Request values come from monitoring production:

1. Deploy with conservative estimates
2. Run under typical load
3. Monitor actual CPU and memory usage
4. Set requests slightly higher than typical usage
5. Iterate as you gather data

Most applications need:

• Web servers: 100-500m CPU, 128-256Mi memory
• Databases: 1000m+ CPU, 1-4Gi memory
• Workers: 500m CPU, 256-512Mi memory

Start with these and adjust based on actual usage.

Common Mistakes

Too high requests: Waste resources and prevent scaling. Too low requests: Cause container starvation and crashes. No limits: Allow misbehaving containers to crash everything. No requests: Allow scheduler to overpack nodes.

Finding the right balance takes monitoring and iteration. The effort pays dividends in stability.

Practice 2: Implement Comprehensive Health Checks

Health checks enable orchestration to detect and fix problems automatically.

Liveness Probes

Liveness probes determine if a container should stay running:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

Every 10 seconds, the orchestrator makes an HTTP request to /health. If the response indicates unhealthy status, the orchestrator restarts the container.

Liveness probes fix containers that are stuck in bad states. The application is running but not functioning properly.

Readiness Probes

Readiness probes determine if a container should receive traffic:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Readiness probes are checked more frequently than liveness probes. A container might be alive but not ready (still starting up, handling a deployment, recovering from failure).

Traffic only routes to containers that are ready. This prevents users from hitting services that haven't finished initialization.

Startup Probes

Startup probes give containers time to start:

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Some containers take 30+ seconds to start (database migrations, cache warming, etc). Startup probes prevent restart loops during initialization.

Implementing Health Endpoints

Your application needs to expose health endpoints:

app.get('/health', (req, res) => {
  if (isPassing()) {
    res.status(200).json({ status: 'healthy' });
  } else {
    res.status(503).json({ status: 'unhealthy' });
  }
});

app.get('/ready', (req, res) => {
  if (dependenciesReady()) {
    res.status(200).json({ ready: true });
  } else {
    res.status(503).json({ ready: false });
  }
});

Health checks aren't about responding to requests. They're about reporting actual application health. A container might respond to HTTP requests but be unable to connect to its database. The health endpoint should return 503 in that case.

Common Mistakes

No health checks: Container crashes go undetected. Too strict health checks: Containers restart unnecessarily. Too lenient health checks: Unhealthy containers keep receiving traffic. Health check timeouts: Container restarts cascade. Missing dependencies in health checks: Checks pass but the container can't function.

Tune health check parameters based on application behavior. Most applications benefit from:

• Liveness probe: 30s initial delay, 10s period, 3 failure threshold
• Readiness probe: 5s initial delay, 5s period, 3 failure threshold
• Startup probe: 30s per attempt, 30 failure threshold for 5 minutes

Practice 3: Configure Network Policies for Security

Network policies implement security within your cluster.

Default Deny Principle

Start by denying all traffic, then explicitly allow what's needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This policy denies all ingress and egress traffic to all pods.

Allow Specific Traffic

Then create policies allowing specific communication:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-db
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 5432

This policy allows traffic from web pods to database pods on port 5432 only.

Benefits

Network policies prevent:

• Lateral movement if a container is compromised
• Accidental exposure of internal services
• Data exfiltration to external sources
• Unwanted communication between services

Implementation Strategy

1. Start with default deny-all policies
2. Document required communication paths
3. Create policies allowing each path
4. Test thoroughly before production deployment
5. Monitor for unexpected policy violations

Without network policies, any compromised container can attack any other container in your cluster. Network policies isolate the blast radius.

Common Mistakes

No network policies: Zero internal security. Too permissive policies: Defeats the purpose of policies. Policies that break applications: Creates operational nightmares. Not monitoring policy violations: Attacks go undetected.

Start restrictive. Open up communication as needed. Monitor violations and refine policies.

Practice 4: Implement Declarative Infrastructure as Code

Declarative configuration is orchestration's strongest feature.

Configuration Files (YAML)

Define your entire infrastructure in version-controlled files:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: myapp:1.2.3
        resources:
          requests:
            cpu: 250m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

This YAML declares that you want 3 replicas of the web application with specific resource limits and health checks.

Benefits of Declarative Configuration

Reproducibility: Same configuration produces the same result everywhere. No manual steps, no variations.

Version Control: Changes are tracked. Rollback to previous configuration is simple.

Auditability: Who changed what and when is visible in version control history.

Disaster Recovery: Infrastructure can be recreated from version-controlled files.

Team Collaboration: Reviewers can comment on infrastructure changes before deployment.

GitOps Workflow

Combine declarative configuration with GitOps:

1. Developers commit infrastructure changes to Git
2. CI/CD pipeline validates changes
3. Approvers review and merge to main branch
4. Deployment system automatically applies changes to production

This workflow means production infrastructure changes only through Git. Everything is audited and reviewed.

Common Mistakes

Manual configuration changes: Creating configuration drift between environments. Configuration stored outside version control: Making changes irrecoverable. No review process: Mistakes reach production. Incomplete configuration: Missing details that should be in files.

Invest in GitOps tooling (ArgoCD, Flux, etc). The small upfront investment pays dividends through reduced errors and better traceability.

Practice 5: Set Up Auto-Scaling Intelligently

Auto-scaling prevents both over-provisioning and under-provisioning.

Horizontal Pod Auto-scaling

Configure the orchestrator to scale container replicas based on metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

This configuration maintains 3-20 replicas, scaling up when CPU exceeds 70% or memory exceeds 80%.

Scaling Behavior

Tune scaling behavior to prevent flapping:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 50
      periodSeconds: 60
  scaleUp:
    stabilizationWindowSeconds: 60
    policies:
    - type: Percent
      value: 100
      periodSeconds: 60

Scale-down stabilization prevents rapidly shrinking when demand temporarily dips. Scale-up happens faster to handle demand spikes.

Custom Metrics

Standard metrics (CPU, memory) don't fit all applications. Custom metrics scale based on application-specific data:

metrics:
- type: Pods
  pods:
    metric:
      name: requests_per_second
    target:
      type: AverageValue
      averageValue: "1000"

E-commerce platforms scale based on orders per second. API services scale based on requests per second. Message queues scale based on queue depth.

Vertical Pod Auto-scaling

VPA adjusts resource requests and limits based on actual usage:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  updatePolicy:
    updateMode: "Auto"

Instead of scaling replicas, VPA scales individual container resources. Use when horizontal scaling isn't appropriate.

Common Mistakes

No auto-scaling: Manual scaling is reactive and slow. Too aggressive scaling: Causes flapping and wasted resources. Scaling on wrong metrics: Doesn't match actual bottlenecks. No minimum replicas: Service becomes unavailable during scale-down. No maximum replicas: Cost explodes during attacks or buggy behavior.

Start with CPU and memory scaling. Add custom metrics once you understand your application's bottlenecks. Monitor and refine scaling policies.

Practice 6: Implement Proper Logging and Monitoring

Visibility is essential. You can't fix what you can't see.

Container Logging

Send all logs to centralized storage:

apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  containers:
  - name: web
    image: myapp:1.2.3
    stdout:
      enabled: true
    stderr:
      enabled: true

Logs should include:

• Request logs (what users are doing)
• Error logs (what went wrong)
• Debug logs (diagnostic information)
• Performance logs (timing information)

Structure logs as JSON for easier parsing:

{
  "timestamp": "2026-09-11T14:30:00Z",
  "level": "error",
  "service": "web-app",
  "message": "Database connection failed",
  "error": "connection timeout",
  "request_id": "abc123"
}

Metrics Collection

Instrument your application to emit metrics:

• Request count and latency
• Error rate
• Database query time
• Cache hit rate
• Custom business metrics

Use Prometheus or similar collectors to scrape metrics.

Monitoring Dashboards

Create dashboards showing:

• Application health
• Resource utilization
• Error rates and types
• Latency percentiles
• Capacity and scaling status

Dashboards should answer questions without requiring log searching.

Alerting

Configure alerts for conditions requiring attention:

• Error rates > 1%
• Latency > 1 second (p99)
• Pod restarts > 3 per hour
• Node CPU > 85%
• Disk space < 10%

Alerts should be specific enough to act on. Generic alerts get ignored.

Common Mistakes

No centralized logging: Logs disappear when pods restart. Unstructured logs: Hard to search and analyze. No metrics: Can't understand application behavior. Too many alerts: Alert fatigue causes critical alerts to be ignored. Storing only recent history: Can't debug issues that happen intermittently.

Invest in logging and monitoring infrastructure. The visibility prevents most operational issues.

Practice 7: Execute Rolling Updates Safely

Deployments should be invisible to users.

Rolling Update Strategy

Default strategy gradually replaces old containers:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

• maxSurge: 1 allows 1 extra pod during update
• maxUnavailable: 0 ensures pods are always available

Process:

1. Start 1 new pod with new version
2. Wait for it to become ready
3. Shift traffic to new pod
4. Terminate 1 old pod
5. Repeat until all pods are updated

Zero downtime and gradual rollout enable easy rollback if issues appear.

Canary Deployments

For higher-risk deployments, deploy to small percentage first:

apiVersion: fluxcd.io/v1beta1
kind: Canary
metadata:
  name: web-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  progressDeadlineSeconds: 60
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5
    metrics:
    - name: error_rate
      thresholdRange:
        max: 0.05

Canary deployments route 5% of traffic to new version. If metrics stay good for several minutes, increase to 50%, then 100%. If metrics degrade, automatic rollback occurs.

Blue-Green Deployments

Deploy new version alongside old version:

1. Deploy new version (green)
2. Test thoroughly in production-like environment
3. Switch all traffic to green
4. Keep blue available for quick rollback

Blue-green deployments enable instant rollback but require double resources temporarily.

Pre-deployment Validation

Before rolling out, validate:

• All dependency services are healthy
• Database migrations have completed
• Configuration is valid
• New image has been tested

Automated pre-deployment checks prevent known issues.

Common Mistakes

No rollback strategy: Problems in production persist. maxUnavailable > 0: Users experience outages during deployment. No health checks: Broken services are deployed. Rollback is manual: Takes too long to recover. No smoke tests: Broken deployments reach production.

Automate the entire deployment pipeline. Include validation, gradual rollout, and automatic rollback.

Practice 8: Plan for Disaster Recovery

Assume everything will fail. Plan accordingly.

Backup Strategy

Backup all persistent data:

• Database backups (hourly or more frequently)
• Configuration backups (version control is usually sufficient)
• User-generated content (synced to backup storage)

Test backups regularly. A backup that can't be restored is worse than no backup.

Backup Retention

Keep multiple backup generations:

• Hourly backups: keep 24 hours
• Daily backups: keep 30 days
• Weekly backups: keep 52 weeks
• Monthly backups: keep 7 years

This strategy enables recovery from various failure points.

Cluster Recovery

Design clusters for recovery:

• Multiple nodes (usually 3+ masters, 3+ workers minimum)
• Persistent volumes distributed across nodes
• Regular cluster backups to separate storage
• Tested recovery procedures

Document and practice cluster recovery regularly.

Multi-region Deployment

For critical applications, replicate across regions:

• Primary region handles all traffic normally
• Secondary region stays in sync
• On primary failure, traffic shifts to secondary
• Recovery takes minutes instead of hours

Multi-region setups increase costs but enable high availability.

RTO and RPO Targets

Define recovery objectives:

• RTO (Recovery Time Objective): How long until service is available
• RPO (Recovery Point Objective): How much data can be lost

Targets guide disaster recovery strategy:

• Website (1 hour RTO, 1 hour RPO): Daily backups sufficient
• E-commerce (15 minute RTO, 5 minute RPO): Continuous replication needed
• Payment system (1 minute RTO, 0 minute RPO): Multi-region active-active setup needed

Testing Disaster Recovery

Regularly test disaster recovery:

• Quarterly full cluster recovery drills
• Monthly backup restoration tests
• Annual multi-region failover tests

The only way to know if disaster recovery works is to test it. Untested recovery plans are fantasies.

Common Mistakes

No backups: Data loss is permanent. Untested backups: Recovery fails when needed. No documentation: Recovery procedure is undiscovered. No team training: Recovery requires specific knowledge. Too long RTO: Business is damaged during recovery.

Treat disaster recovery as seriously as the application itself. Invest in tooling and testing. The downtime you prevent could save your company.

Key Takeaways

• Resource requests and limits prevent container starvation and runaway consumption through proper scheduler placement and runtime enforcement
• Comprehensive health checks (liveness, readiness, startup) enable automatic container healing and prevent traffic routing to broken containers
• Network policies implement zero-trust networking, preventing lateral movement if containers are compromised
• Declarative infrastructure as code enables reproducibility, auditability, and easy disaster recovery through version control
• Intelligent auto-scaling based on metrics prevents over-provisioning and under-provisioning, optimizing cost and performance
• Centralized logging and monitoring provide visibility into application behavior and enable proactive issue detection
• Rolling updates with health checks and automated rollback ensure zero-downtime deployments
• Comprehensive disaster recovery planning with tested procedures enables recovery from complete infrastructure failure

Frequently Asked Questions

Q: How often should we test disaster recovery? 

A: Quarterly full cluster recovery drills are standard. Critical systems should test monthly. Teams often find issues only during actual tests.

Q: What's the minimum logging you need? 

A: Error logs, request logs (request ID, path, latency), and health check results. Start there and expand based on what you need to debug.

Q: How do we choose between blue-green and rolling updates? 

A: Rolling updates are standard for most applications. Blue-green deployments are safer for high-risk applications but use double resources. Canary deployments offer middle ground.

Q: What's a reasonable RTO for most applications? 

A: 15-60 minutes for most web applications. 5-15 minutes for revenue-generating systems. 1-5 minutes for critical infrastructure.

Q: Should we scale to zero replicas? 

A: Only for non-critical workloads. Keep at least 1 replica for important services. Scaling to zero risks service unavailability.

Q: How do we implement multi-region deployment? 

A: Start with primary-secondary setup. Automated failover adds complexity. Consider managed services (AWS Global Accelerator, etc) for easier implementation.

Q: What monitoring metrics matter most? 

A: Error rate, latency (p50/p99), request volume, CPU/memory utilization, and pod restart rate. Business metrics (revenue, transactions) are equally important.

Q: How do we prevent alert fatigue? 

A: Only alert on actionable conditions. Set thresholds high enough to avoid noise. Separate alerts by severity. Route severe alerts to on-call, others to team channel.

Q: What's typical auto-scaling response time? 

A: Detection takes 1-2 minutes. Scaling takes 30 seconds to several minutes depending on container startup time. Total: 2-5 minutes typical.

Q: Should we implement network policies immediately? A: No, implement after you understand communication patterns. Start with allow-all, then restrict based on documented needs.

Conclusion

Container orchestration is powerful. With power comes responsibility.

These eight practices separate production systems you trust from systems that keep you up at night.

Implementing all eight at once is ambitious. Start with health checks and resource limits. Add network policies when security becomes important. Gradually implement logging, auto-scaling, rolling updates, and disaster recovery.

The journey to production-grade orchestration takes time. Each practice improves reliability. Together, they enable 99.99% uptime with minimal manual intervention.

The effort pays dividends: less firefighting, faster feature delivery, and confidence that your infrastructure can handle failure.

Ready to understand how containerization and orchestration fit together in your infrastructure strategy?

For platform-specific implementation guidance, refer to official documentation and consider consulting deployment guides tailored to your use case.