The Complete Responsive CMS Blog created by Francesco Malagrino

Seeing Through the Noise: Deploying a Turnkey Observability Platform for Distributed Systems

Category: Systems Architecture & Written by Francesco Malagrino On July-15-2026 14:44:17

---

Introduction

Moving from a monolithic application to distributed microservices can improve scalability, deployment independence and fault isolation. It also creates a significant operational challenge: a single user request may cross an API gateway, several services, a Kafka event stream, a saga orchestrator and multiple databases before completing.

When something fails, application logs from one container rarely explain the entire problem. Engineers need to understand which service failed, where latency was introduced, which event triggered a compensation flow and whether the issue is isolated or affecting the wider platform.

The observability-platform-turnkey project provides a deployable observability architecture built around OpenTelemetry, Prometheus, Loki, Tempo, Grafana, Alertmanager and Kong. It centralises telemetry from distributed services and makes metrics, logs and traces available through one operational platform.

The Architectural Challenge

Observability is not simply the installation of a dashboard.

A complete platform must collect telemetry from different applications, process it consistently, store each signal in an appropriate backend and preserve the context needed to correlate an error across service boundaries.

The architecture therefore separates several responsibilities:

• Kong manages API traffic, authentication, rate limiting and correlation context.


• Instrumented services emit telemetry through OpenTelemetry.


• The OpenTelemetry Collector receives, processes and distributes telemetry.


• Prometheus stores metrics and evaluates alerting rules.


• Loki aggregates structured logs.


• Tempo stores distributed traces.


• Grafana provides dashboards and cross-signal investigation.


• Alertmanager routes operational alerts according to severity.

This creates a common observability layer for the microservices, saga and multi-cloud projects without forcing every application to communicate directly with multiple monitoring backends.

Kong as the Observable Entry Point

Kong acts as the API gateway at the edge of the architecture.

Incoming traffic passes through a consistent control point that supports routing, JWT authentication, rate limiting, OpenTelemetry tracing and correlation IDs. This provides an initial request context that can be propagated through downstream services.

The gateway also exposes operational signals such as upstream health, request rates, latency and error responses. These signals help engineers determine whether a problem originates at the edge, inside an application service or farther downstream.

OpenTelemetry as the Telemetry Hub

The OpenTelemetry Collector is the central telemetry pipeline.

Applications send telemetry through the standard OTLP protocol over gRPC or HTTP. The collector then applies processing stages such as batching, tail sampling and span-metric generation before exporting the resulting signals to the appropriate storage systems.

This architecture decouples application instrumentation from backend-specific implementations. Services emit telemetry through OpenTelemetry rather than being tightly coupled to Prometheus, Tempo or another vendor-specific platform.

The repository also includes a Node.js and TypeScript instrumentation example using the OpenTelemetry SDK and automatic instrumentation packages. Applications still require appropriate instrumentation, but the collector provides one consistent ingestion and processing layer.

The Three Pillars of Observability

Metrics

Prometheus collects time-series data from applications, infrastructure components and Kubernetes environments.

The platform focuses on signals that help teams evaluate both technical health and business-process behaviour, including:

• Request rate


• Error rate


• p99 latency


• Kafka consumer lag


• Dead-letter queue depth


• Kong upstream health


• Saga completion and failure rates


• Compensation activity


• Kubernetes and infrastructure availability

Prometheus rules convert these signals into operational alerts. Examples include elevated HTTP 5xx rates, excessive p99 latency, failed saga compensation, unavailable telemetry components and messages accumulating in a Kafka dead-letter queue.

Logs

Loki and Promtail provide centralised log aggregation.

Promtail collects container and application logs and forwards them to Loki, where they are organised through labels such as service, namespace and environment. Grafana then allows engineers to query logs without connecting directly to individual containers or Kubernetes nodes.

The platform also supports correlation between logs and traces. When applications include trace identifiers in structured logs, engineers can move from a failing trace to the related log entries and investigate the same request across multiple services.

Unlike traditional full-text search platforms, Loki primarily indexes labels rather than the complete contents of every log line. This can reduce indexing overhead, but it also requires a deliberate labelling and structured-logging strategy.

Distributed Tracing

Tempo stores distributed traces produced through OpenTelemetry instrumentation.

A trace records the sequence of operations involved in processing a request. Individual spans show which services participated, how long each operation took and where an error occurred.

This is particularly valuable for asynchronous and saga-based architectures. A single business transaction may involve an API request, Kafka events, payment processing, inventory changes and compensation logic.

Tracing provides the context needed to reconstruct that flow instead of investigating each component independently.

Purpose-Built Grafana Dashboards

The repository provisions Grafana data sources and dashboards automatically.

The Services Overview dashboard presents request rates, error rates, p99 latency, Kafka lag, dead-letter queue depth and Kong upstream health.

A dedicated Saga State Machine dashboard displays:

• Sagas started, completed and failed


• Compensation rates


• p99 saga duration


• Transactions stuck in COMPENSATION_FAILED


• Related log entries

These dashboards are not intended to replace deeper investigation. They provide a common operational starting point from which engineers can move between metrics, logs and traces.

Alerting and Incident Prioritisation

Prometheus evaluates predefined alerting rules, while Alertmanager handles routing and prioritisation.

Critical alerts are delivered immediately. Warning alerts are grouped to reduce unnecessary notification volume. Inhibition rules suppress lower-severity warnings when a critical alert is already active for the same service.

The repository includes alerts for conditions such as:

• High HTTP 5xx error rates


• Excessive p99 latency


• Saga compensation failures


• OpenTelemetry Collector outages


• Kafka dead-letter queue accumulation


• Unreachable Prometheus targets


• Loki or Tempo outages

This transforms the platform from a passive dashboard into an active operational monitoring system.

Local and Multi-Cloud Deployment

The platform supports multiple deployment models.

For local development and demonstrations, Docker Compose starts the observability components and exposes Grafana as the primary interface. A debug overlay can temporarily expose Prometheus, Tempo and Alertmanager when direct troubleshooting is required.

For Kubernetes environments, the repository provides deployment blueprints for:

• Amazon EKS


• Google GKE


• Azure AKS

Provider-specific installation scripts configure the platform for each environment. A generic installer also supports selectable cloud providers, authentication modes and sizing profiles.

The available profiles cover smaller development environments, medium production deployments with fewer than 50 services and larger production environments with additional replicas and longer Prometheus retention.

Cloud-Native Storage and Authentication

Loki and Tempo require durable object storage for production deployments.

The architecture supports:

• Amazon S3 on AWS


• Google Cloud Storage on GCP


• Azure Blob Storage on Azure


• MinIO for local development

Production environments use cloud-native workload identities rather than embedded static access keys. AWS uses IAM Roles for Service Accounts, while GCP and Azure use workload identity integrations.

Terraform modules provision the required storage and identity resources for each cloud provider.

This keeps telemetry data within the selected cloud environment while preserving a consistent platform architecture across EKS, GKE and AKS.

Security by Default

An observability platform processes sensitive operational information and must be protected like any other production system.

The Kubernetes deployment includes several security controls:

• Default-deny NetworkPolicies with explicit communication rules


• mTLS between the OpenTelemetry Collector and backend components


• Certificates managed through cert-manager


• Kubernetes Secrets and External Secrets Operator support


• Cloud-native workload authentication


• No public exposure of the Kong Admin API

These controls reduce unnecessary network access and avoid relying on static credentials inside observability workloads.

Backup, Disaster Recovery and Operations

The observability platform is itself a critical production dependency. Losing it during an incident would remove the evidence needed to diagnose the incident.

The repository uses Velero to back up Kubernetes resources and persistent volumes. It also includes a restore-drill script that verifies whether backups can actually be recovered.

Operational runbooks cover:

• Backup and restoration


• Disaster recovery


• Incident response


• Capacity management


• Data retention


• Platform upgrades

This is an important distinction between installing monitoring tools and operating an observability platform as a production service.

Limitations and Trade-Offs

A turnkey platform does not make every application observable automatically.

Services must still emit meaningful metrics, propagate trace context and produce structured logs. Poor instrumentation will create attractive dashboards without providing enough information to diagnose real failures.

Telemetry also introduces infrastructure costs. High-cardinality metrics, excessive logging and unsampled traces can consume significant storage and processing capacity. Retention, sampling and labelling policies must therefore be designed deliberately.

The observability platform must also be monitored independently. Teams need to detect when Prometheus stops scraping, when the OpenTelemetry Collector drops data or when Loki and Tempo become unavailable.

Finally, telemetry may contain customer identifiers, request data or sensitive operational information. Access controls, retention policies and data-scrubbing rules must be applied before production deployment.

Conclusion

Observability is the ability to understand a system through the signals it produces.

By combining Kong, OpenTelemetry, Prometheus, Loki, Tempo, Grafana and Alertmanager, this platform provides a unified foundation for investigating distributed applications across metrics, logs and traces.

Its value goes beyond centralised dashboards. The architecture includes multi-cloud deployment support, cloud-native authentication, network isolation, mTLS, automated alerting, backup procedures, restore testing and operational runbooks.

The result is a practical observability blueprint that helps engineers move from symptoms to root causes with evidence rather than guesswork.

Explore the Implementation

Review the OpenTelemetry pipeline, Prometheus rules, Grafana dashboards, Loki and Tempo configurations, Kubernetes deployment profiles, Terraform modules, installation scripts and operational runbooks in the repository: Observability Platform Turnkey

Facebook Whatsapp Twitter Linkein Pinterest Reddit Tumblr Mail

Systems Architecture
Cloud Native Architecture
Site Reliability Engineering (SRE)
typescript
nodejs
Distributed-Systems
Microservices
Software Architecture
Rest
Java
Technology

---

---

---

---

---