Observability

Monitoring tells you that something is wrong (a metric crossed a threshold, an alert fired). Observability is what lets you actually figure out why — by giving you enough raw, queryable signal about the system's internal state to debug a problem you didn't anticipate in advance. The three pillars below are the standard decomposition of what that raw signal is made of.

Logs

Structured, timestamped records of discrete events — a request came in, a model version loaded, an exception was thrown. "Structured" is the operative word for production systems: unstructured free-text logs are fine to grep by hand, but structured logs (JSON, with consistent fields) are what makes logs queryable and aggregatable at scale.

• ELK stack (Elasticsearch, Logstash, Kibana): the classic self-hosted logging stack — Logstash ingests and processes logs, Elasticsearch indexes and stores them, Kibana visualizes and queries them.
• OpenSearch: an open-source fork of Elasticsearch (post-license-change), largely API-compatible, increasingly the default when avoiding Elastic's licensing terms matters.
• Loki: a log aggregation system built by Grafana Labs, designed to be cheaper to run than Elasticsearch by indexing only metadata (labels) rather than full log content — pairs naturally with Grafana/Prometheus.

Metrics

Numeric measurements aggregated over time — request rate, error rate, latency, GPU utilization, model prediction distribution. Unlike logs, metrics are cheap to store at high volume because they're pre-aggregated, which is exactly what makes them good for dashboards and alerting rather than deep forensic debugging.

• Prometheus: the standard open-source metrics collection system — services expose a /metrics endpoint, Prometheus scrapes it on an interval and stores a time series; its query language (PromQL) is what most alerting rules and dashboards are built on.
• Grafana: the standard visualization layer on top of Prometheus (and Loki, and most other data sources) — dashboards, alerting, and exploration, decoupled from whichever backend actually stores the data.

Instrumenting the FastAPI serving endpoint from APIs & Model Serving to expose exactly the metrics a /metrics scrape needs:

from prometheus_client import Counter, Histogram, make_asgi_app

PREDICTIONS = Counter("model_predictions_total", "Total predictions served", ["model_version"])
LATENCY = Histogram("model_predict_latency_seconds", "Prediction latency")

app.mount("/metrics", make_asgi_app())  # Prometheus scrapes this

@app.post("/v1/predict")
@LATENCY.time()
def predict(req: PredictRequest):
    PREDICTIONS.labels(model_version="3").inc()
    return PredictResponse(prediction=model.predict([req.features])[0])

The PromQL a dashboard or alert would actually run against that data — p95 latency, and an alert rule on error rate:

# p95 prediction latency over the last 5 minutes
histogram_quantile(0.95, rate(model_predict_latency_seconds_bucket[5m]))

groups:
  - name: model-api
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 10m
        annotations:
          summary: "model-api error rate above 5% for 10 minutes"

Traces

A trace follows a single request across every service it touches, recording how long each step took — essential the moment a system is more than one service, because "the API was slow" could mean the API, the feature store, the model server, or a downstream database, and only a trace shows which one.

• OpenTelemetry: the current standard, vendor-neutral instrumentation framework for generating traces (and metrics, and logs) — instrument once, export to whichever backend you choose.
• Jaeger: a common open-source backend for storing and visualizing traces, frequently paired with OpenTelemetry instrumentation.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/v1/predict")
def predict(req: PredictRequest):
    with tracer.start_as_current_span("feature-lookup"):
        features = fetch_features(req.user_id)
    with tracer.start_as_current_span("model-inference"):
        prediction = model.predict([features])[0]
    return PredictResponse(prediction=prediction)

Why All Three, Not Just One

Each pillar answers a different question a real incident needs: metrics tell you something degraded and roughly when (a dashboard spike), traces tell you where in a multi-service request path it happened, and logs tell you exactly what happened at that point (the specific error, the specific input). Alerting is usually built on metrics (cheap, fast to query); root-causing an actual incident usually means pivoting from a metric spike into the traces and logs around that same time window.

Next: GPU/AI Infrastructure & Distributed Training — the hardware layer everything above is ultimately monitoring.