The Complete Responsive CMS Blog created by Francesco Malagrino

Architecting for Failure: Building a Turnkey Chaos Engineering Platform on Kubernetes

Category: Site Reliability Engineering (SRE) & Written by Francesco Malagrino On July-15-2026 14:35:10

---

Introduction

In distributed systems, hoping that resilience mechanisms will work during an outage is not a strategy. High availability, retries, compensation flows and recovery procedures must be tested under controlled failure conditions before they are needed in production.

This project provides a structured chaos engineering platform for Kubernetes. It combines Chaos Mesh for fault injection, LitmusChaos for workflow orchestration and resilience scoring, Prometheus for steady-state validation, and Grafana for visualising experiment results.

The objective is not to break systems randomly. It is to introduce measurable, tightly controlled failures and verify that the platform responds as designed.

The Core Architecture

The platform separates fault injection, orchestration and observability into distinct responsibilities.

Chaos Mesh: Fault Injection

Chaos Mesh provides declarative Kubernetes Custom Resource Definitions for introducing controlled failures.

The repository contains ten experiments covering six fault categories:

• Network latency, packet loss and service partitions


• Pod termination and simulated pod failure


• CPU and memory pressure


• PostgreSQL disk I/O latency


• Clock skew


• HTTP error injection

Each experiment is defined as YAML and can be reviewed, versioned and executed through standard Kubernetes workflows.

LitmusChaos: Game Day Orchestration

LitmusChaos coordinates the wider resilience game day rather than replacing Chaos Mesh.

The workflow sequences experiments, evaluates Prometheus probes and calculates a resilience score between 0 and 100. This provides a repeatable measurement of how the system behaves across releases instead of relying only on manual observations.

The target defined by the project is a resilience score of at least 80 before a production release.

Prometheus and Grafana: Observation and Evidence

Before an experiment begins, the platform evaluates a steady-state hypothesis.

The checks verify conditions such as:

• Order and payment service success rates


• Order-service p99 latency


• Kubernetes pod readiness


• Argo Rollouts health


• ArgoCD synchronisation

If the platform is already degraded, the experiment should not proceed. Running chaos against an unhealthy system would make the results unreliable and could worsen an existing incident.

During and after the experiment, Grafana displays service health, latency, experiment verdicts and resilience-score trends.

Fault Injection as Code

Chaos experiments are treated as version-controlled infrastructure artifacts rather than one-off operational scripts.

For example, an experiment can introduce 200 milliseconds of latency into the order service, drop Kafka packets, terminate an application pod, create CPU pressure on the payment service or inject HTTP 503 responses into a percentage of order requests.

Because these experiments are declarative, they can be reviewed through pull requests, validated in CI and reproduced consistently during future game days.

Blast-Radius Control

A chaos engineering platform is only useful when failures remain controlled.

Experiments use namespaces, label selectors, duration limits and explicit risk classifications to constrain their impact. The repository categorises experiments as low, medium or high risk.

High-risk experiments are restricted to staging, and CI validates that they cannot target the production microservices namespace. The platform also includes an emergency command for removing active experiments immediately.

Failure Walkthrough: Payment-Service Network Partition

Consider a scenario in which the payment service becomes completely unreachable while the rest of the platform remains operational.

Pre-Experiment Validation

Before introducing the fault, the safe experiment runner queries the steady-state conditions.

The system must be healthy, services must meet their expected success-rate and latency thresholds, pods must be ready, and no active deployment or incident should already be affecting the environment.

Injection

Chaos Mesh applies a NetworkChaos resource to one payment-service pod in the staging namespace.

The experiment cuts both ingress and egress traffic for two minutes, simulating a complete pod-level network failure rather than an entire availability-zone outage.

Observation

Prometheus and Grafana expose the resulting reduction in service availability and request success.

The observability platform should identify the payment-service failure, while Alertmanager is expected to raise the relevant service-down alert. Deployment and application health can also be observed through ArgoCD and the Kubernetes platform.

Validation

The primary validation is whether the surrounding distributed system behaves safely.

The saga orchestrator must detect that payment processing failed and trigger the appropriate compensation path. The test must also confirm that payment operations are neither duplicated nor silently lost.

After the fault is removed, the platform verifies that the service returns to its original steady state and records the experiment outcome in the resilience score.

Safe Experiment Execution

The repository includes a safe runner that performs steady-state checks before and after every experiment.

This creates a controlled lifecycle:

1. Confirm that the system is healthy.


2. Apply the chaos experiment.


3. Observe system behaviour.


4. Remove or allow the fault to expire.


5. Verify recovery.


6. Generate the resilience result.

The project also includes an experiment-failure runbook, a production-incident runbook and a structured game-day playbook so that chaos testing is supported by operational procedures rather than YAML files alone.

Production Safety and Guardrails

Chaos engineering should progress gradually.

Low-risk experiments can be automated regularly in staging. Medium-risk experiments are appropriate before significant releases, while high-risk experiments require manual execution with an engineer present and are restricted to staging by the current platform design.

Production experiments should only be considered after the platform has reliable monitoring, distributed tracing, alerting, service-level objectives, ownership procedures and an immediate mechanism for stopping active faults.

Teams must also be able to distinguish injected failures from genuine incidents. If an actual outage begins during a chaos experiment, the injected fault should be stopped immediately so that responders can isolate the real cause.

A staging environment cannot reproduce production perfectly, but it should represent production topology, configuration and traffic patterns closely enough to generate meaningful resilience evidence.

Local and Kubernetes-Based Testing

The project supports two testing models.

A Docker Compose environment provides mock services and Grafana for local demonstrations without requiring Kubernetes. A kind-based setup supports real Kubernetes fault injection using Chaos Mesh and LitmusChaos.

This allows engineers to validate the platform locally before connecting it to managed Kubernetes environments such as EKS, AKS or GKE.

Conclusion

Chaos engineering is not the uncontrolled destruction of infrastructure. It is a disciplined method for testing assumptions about system resilience.

By combining declarative fault injection, steady-state validation, controlled blast radii, automated scoring, observability and operational runbooks, this platform turns resilience testing into a repeatable engineering process.

The result is a practical way to identify weaknesses before real outages expose them—and to replace assumptions about recovery with measurable evidence.

Explore the Implementation

Review the Chaos Mesh experiments, LitmusChaos game-day workflow, Prometheus steady-state checks, Grafana dashboard, automation scripts and operational runbooks in the repository: Chaos Engineering Platform

Facebook Whatsapp Twitter Linkein Pinterest Reddit Tumblr Mail

Systems Architecture
Cloud Native Architecture
Site Reliability Engineering (SRE)
typescript
nodejs
Distributed-Systems
Microservices
Software Architecture
Rest
Java
Technology

---

---

---

---

---