The Complete Observability Guide

When production systems break, observability determines whether you find the root cause in minutes or hours. Monitoring tells you that something is wrong. Observability lets you ask arbitrary questions about your system's behavior without deploying new instrumentation. This guide covers how to implement the three pillars of observability and connect them into a coherent debugging workflow.

The Three Pillars

Observability rests on three complementary signal types:

• Metrics: Numeric measurements aggregated over time. "Request latency p99 is 450ms." Metrics tell you what is happening at a macro level and are the foundation of alerting.
• Traces: Records of requests as they flow through distributed systems. "This request spent 200ms in the API, 150ms in the database, and 80ms in the cache." Traces tell you where time is spent.
• Logs: Discrete events with contextual data. "User abc123 failed authentication: invalid password." Logs tell you why something happened.

Each pillar answers different questions. Metrics detect problems. Traces locate problems. Logs explain problems. The power comes from correlating all three: an alert fires on a latency metric, you find a slow trace, and the trace leads you to a log entry showing a database timeout.

OpenTelemetry Setup

OpenTelemetry (OTel) provides a vendor-neutral standard for collecting metrics, traces, and logs. Instrument once, export to any backend.

// tracing.ts - Initialize OpenTelemetry for a Node.js service
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'payment-service',
    [ATTR_SERVICE_VERSION]: '1.2.0',
    environment: process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://otel-collector:4318/v1/metrics',
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
    }),
  ],
});

sdk.start();

Auto-instrumentation captures HTTP requests, database queries, and framework-specific spans without modifying application code. Add manual instrumentation for business-specific operations:

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service');

async function processPayment(order: Order): Promise<PaymentResult> {
  return tracer.startActiveSpan('process-payment', async (span) => {
    span.setAttribute('order.id', order.id);
    span.setAttribute('order.amount', order.amount);
    span.setAttribute('order.currency', order.currency);

    try {
      const result = await chargeCard(order);
      span.setAttribute('payment.status', result.status);
      return result;
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Distributed Tracing

In microservice architectures, a single user request might touch dozens of services. Distributed tracing connects the spans from each service into a single trace by propagating a trace context through headers.

Trace: abc-123
├── [API Gateway] GET /orders/456           (12ms)
│   ├── [Auth Service] validate-token       (3ms)
│   ├── [Order Service] get-order           (45ms)
│   │   ├── [PostgreSQL] SELECT orders      (8ms)
│   │   └── [Cache] redis.get              (1ms)
│   └── [Inventory Service] check-stock     (22ms)
│       └── [DynamoDB] GetItem              (15ms)

The trace context propagates automatically through HTTP headers when using OTel instrumentation:

traceparent: 00-abc123def456789-span456789-01

Each service reads the incoming trace context, creates child spans, and propagates the context to downstream calls. This happens automatically with OTel's HTTP instrumentation.

When debugging with traces:

1. Find a trace that exhibits the problem (slow, error, unexpected behavior).
2. Examine the span waterfall to identify which service or operation is slow.
3. Look at span attributes for context (user ID, request parameters, error messages).
4. Correlate with logs using the trace ID.

Prometheus Metrics

Prometheus uses a pull-based model: it scrapes metrics endpoints at regular intervals. Four metric types cover most use cases:

from prometheus_client import Counter, Histogram, Gauge, Summary

# Counter: monotonically increasing value
requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status_code']
)

# Histogram: distribution of values (latency, sizes)
request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

# Gauge: value that goes up and down
active_connections = Gauge(
    'active_connections',
    'Number of active connections',
    ['service']
)

# Usage in request handler
@app.middleware('http')
async def metrics_middleware(request, call_next):
    start_time = time.time()
    response = await call_next(request)
    duration = time.time() - start_time

requests_total.labels(
    method=request.method,
    endpoint=request.url.path,
    status_code=response.status_code
).inc()

request_duration.labels(
    method=request.method,
    endpoint=request.url.path
).observe(duration)

return response

Choose histogram buckets based on your SLOs. If your latency target is 200ms, you need buckets around that boundary to measure accurately.

## Structured Logging

Structured logs are JSON objects instead of free-text strings. They enable querying, filtering, and correlation across services:

```python
import structlog

logger = structlog.get_logger()

# Configure structured logging
structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ],
)

# Bind context that persists across log calls
structlog.contextvars.bind_contextvars(
    trace_id=span_context.trace_id,
    user_id=request.user_id,
    request_id=request.headers.get('x-request-id'),
)

logger.info("payment_processed",
    order_id="order-456",
    amount=99.99,
    currency="USD",
    payment_method="card",
    processing_time_ms=145,
)

Output:

{
  "event": "payment_processed",
  "level": "info",
  "timestamp": "2024-11-15T14:23:45.123Z",
  "trace_id": "abc123def456",
  "user_id": "user-789",
  "request_id": "req-012",
  "order_id": "order-456",
  "amount": 99.99,
  "currency": "USD",
  "payment_method": "card",
  "processing_time_ms": 145
}

The trace_id field connects this log entry to a distributed trace, enabling you to jump from a log entry to the full request trace and back.

Alerting Strategies

Alerts should notify you of problems that require human intervention. Everything else is noise that erodes on-call morale and causes alert fatigue.

Alert on symptoms, not causes. Alert on "error rate exceeds 1%" rather than "database CPU above 80%." High CPU might not affect users. A high error rate definitely does.

Use multiple severity levels:

• Critical: User-facing impact right now. Pages the on-call engineer.
• Warning: Degradation that will become critical if not addressed. Creates a ticket.
• Info: Notable events for awareness. Appears in a dashboard.

Effective alert rules with PromQL:

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
          > 0.01
        for: 5m  # Must persist for 5 minutes to fire
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 1%"
          dashboard: "https://grafana.example.com/d/api-overview"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency above 1 second"

SLOs and SLIs

Service Level Objectives (SLOs) define the reliability targets your service aims to meet. Service Level Indicators (SLIs) are the metrics that measure performance against those targets.

SLI: The proportion of successful HTTP requests (status < 500)
     measured over a rolling 30-day window.

SLO: 99.9% of requests succeed (error budget: 0.1%)

Error budget: In a 30-day month with 10 million requests,
              0.1% = 10,000 allowed failures.

Track error budget consumption to make informed decisions:

• Budget available: Ship features, run experiments, take calculated risks.
• Budget depleting fast: Slow down releases, prioritize reliability work.
• Budget exhausted: Freeze deployments, focus entirely on stability.

Error budgets transform reliability from "be as reliable as possible" into "be reliable enough and spend the rest of the budget on shipping features." This alignment between development velocity and operational stability is what makes SLOs practical.

Observability is not a tool you buy or a library you install. It is a property of your system that emerges from thoughtful instrumentation, correlated signals, and disciplined alerting. Start with structured logging and basic metrics. Add distributed tracing when you have multiple services. Define SLOs when you need to balance reliability with development speed. Each layer builds on the previous one, and together they give you the ability to understand your system's behavior under any conditions.