Metrics and Monitoring

Metric types, the RED and USE methods, Prometheus, Grafana, and how to measure the behavior of distributed systems.

What are metrics?

Metrics are numeric values that represent the state or behavior of a system at a given point in time. Unlike logs (which capture individual events), metrics capture trends and let you answer questions like: “How many requests per second are we processing?” or “What is the 99th percentile latency?”

Metric types

Counters

A counter is a value that can only increase. It resets when the service restarts.

• Use: Counting cumulative events.
• Examples: Total requests received, total errors, total messages processed.
• Key point: What matters is not the absolute value but the rate of change (requests/second).

Gauges

A gauge is a value that can go up and down. It represents the current state of something.

• Use: Measuring instantaneous values.
• Examples: Active connections, memory usage, queue size, CPU temperature.
• Key point: Useful for detecting saturation and trends.

Histograms

A histogram groups observations into predefined buckets and computes distributions.

• Use: Measuring value distributions, especially latencies.
• Examples: Request latency (p50, p95, p99), payload sizes.
• Key point: Percentiles are more useful than averages — an average of 100ms can hide the fact that 1% of requests take 5 seconds.

Summaries

Similar to histograms, but they compute percentiles on the client instead of the server.

• Use: When you need exact percentiles without configuring buckets.
• Trade-off: They cannot be aggregated across instances, unlike histograms.

Methods for choosing metrics

The RED method (for services)

The RED method focuses on user-oriented metrics for request-driven services:

RED is ideal for microservices that handle HTTP or gRPC requests. If these three metrics are healthy, the service is probably healthy.

The USE method (for resources)

The USE method focuses on the system’s infrastructure and resources:

USE is ideal for diagnosing infrastructure problems: CPU, memory, disk, network, database connections.

Combining RED and USE

In practice, use both methods:

• RED to understand the user experience and the health of services.
• USE to understand the health of the underlying infrastructure.

Prometheus

Prometheus is the de facto standard for metrics in cloud-native architectures.

Pull model

Unlike other systems that receive metrics (push), Prometheus actively collects them (pull). Each service exposes a /metrics endpoint that Prometheus scrapes periodically.

Metrics format

# HELP http_requests_total Total de requests HTTP recibidas
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/api/orders",status="200"} 15234
http_requests_total{method="POST",path="/api/orders",status="201"} 3421
http_requests_total{method="POST",path="/api/orders",status="500"} 12

PromQL

Prometheus includes a powerful query language (PromQL) for analyzing metrics:

# Tasa de requests por segundo en los últimos 5 minutos
rate(http_requests_total[5m])

# Porcentaje de errores
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Latencia del percentil 99
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Alerting with Prometheus

Prometheus integrates with Alertmanager to define alerting rules based on PromQL:

groups:
  - name: servicio-ordenes
    rules:
      - alert: AltaTasaDeErrores
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Alta tasa de errores en el servicio de órdenes"

Grafana

Grafana is the most popular visualization tool for Prometheus metrics.

Effective dashboards

A good Grafana dashboard for a microservice includes:

1. Top panel: RED metrics (rate, errors, duration) as the primary indicators.
2. Latency panels: Histograms with p50, p95, p99 percentiles.
3. Resource panels: CPU, memory, connections (USE metrics).
4. Dependency panels: Latency and errors toward downstream services.

Variables and templates

Grafana lets you create dynamic dashboards with variables:

• A service selector to reuse the same dashboard.
• A time-range selector.
• Filters by instance, HTTP method, or endpoint.

Best practices

Consistent naming

Use clear conventions for naming metrics:

• Prefix with the service name: order_service_requests_total.
• Suffix with the unit: _seconds, _bytes, _total.
• Labels for dimensions: method, status, endpoint.

Keeping cardinality under control

Each unique combination of labels creates a time series. Avoid high-cardinality labels (like user IDs or request IDs) — they can blow up Prometheus storage.

Business metrics

Don’t limit yourself to technical metrics. Business metrics are just as valuable:

• Orders created per minute.
• Successful vs rejected payments.
• Average checkout time.

Golden signals

Google defines four “golden signals” that every service should monitor:

1. Latency: The time it takes to serve requests.
2. Traffic: The volume of requests.
3. Errors: The rate of failed requests.
4. Saturation: How “full” the service is.

Summary

Metrics are the second pillar of observability. While logs tell you what happened, metrics tell you how the system is doing right now and where it’s headed. By combining the RED and USE methods with tools like Prometheus and Grafana, you can build a monitoring system that alerts you before problems reach your users.