← Back to Home
Home/Zipkin/Analysing Latency & Debugging with Zipkin
🔬

Analysing Latency & Debugging with Zipkin

Turn traces into answers — find the slow hop, the hidden retry, the chatty dependency.

⏱10 min read📚DevOps Fundamentals

Traces are flowing — now let’s use them. The Zipkin UI is where you go from “checkout is slow” to “payments is making three retries to the bank API.” This lesson covers reading traces, spotting common problems, and the dependency graph.

🎯 Learning Objectives

Searching for Traces

The Zipkin UI lets you filter traces by service, span name, tags, duration, and time range. To hunt latency, sort by longest duration:

text
Search filters:
serviceName = checkout
minDuration = 500ms          # only slow requests
tagQuery    = error=true     # or only failing ones
lookback    = 1h

This surfaces the exact slow or failing requests — not averages, but real individual journeys you can open and inspect.

💡 Start from a symptom

Don’t browse traces at random. Start from a real complaint — “checkout p95 spiked at 2pm” — filter to that service, time, and a high minDuration, and open the worst offenders.

Reading the Waterfall: The Critical Path

Open a trace and read the waterfall top-to-bottom. The critical path is the chain of spans that actually determines total duration.

text
GET /checkout                        820ms  ◀ total
├─ orders-service                    640ms
│  ├─ SELECT orders                   40ms
│  └─ payments-service               560ms  ◀ dominates
│     └─ POST bank-api  (x3 retries) 520ms  ◀ root cause
└─ inventory-service                  90ms  (parallel, not on critical path)

Reading this: inventory ran in parallel and is cheap; the real cost is payments → bank-api, which is retrying three times. The fix isn’t “optimize checkout” — it’s “why is the bank API failing and retrying?”

Common Problems Traces Reveal

Pattern in the waterfall Likely problem
Many tiny identical spans in a loop N+1 queries — batch them
Same call repeated 3× back-to-back Retries on a failing dependency
Long spans stacked sequentially Serial calls that could run in parallel
A big gap between spans Time lost in queue/GC/lock, not in a call
A span tagged error=true A failure — open it for the exception detail

⚠ A gap is a clue too

Empty time between spans (where nothing is running) often means waiting — on a lock, a thread pool, a queue, or garbage collection. Don’t just look at the bars; look at the gaps.

The Dependency Diagram

Zipkin aggregates traces into a service dependency graph — who calls whom, and how often. It’s invaluable for understanding an unfamiliar system:

text
gateway ──▶ orders ──▶ payments ──▶ bank-api
              │
              └────▶ inventory ──▶ warehouse-db

Use it to spot surprises: a service calling one it shouldn’t, an unexpectedly chatty edge, or a single dependency that everything funnels through (a availability risk).

A Debugging Workflow

Put it together into a repeatable loop:

  1. Alert fires (from your metrics) — latency or errors up on checkout.
  2. Search Zipkin for slow/failing checkout traces in that window.
  3. Open the worst trace, follow the critical path to the dominant span.
  4. Inspect that span’s tags and any error, or spot the retry/N+1 pattern.
  5. Check logs for that exact traceId to get line-level detail.
  6. Fix and verify — after deploying, new traces should show the span shrink.

💡 Metrics + traces + logs together

This is observability in practice: metrics detected it, the trace localized it, and logs (found by traceId) gave the detail. Each pillar hands off to the next.

🧪 Hands-on Lab

📝

Diagnose from the Trace

A user reports slow product pages. This trace shows the problem — what is it, and what’s your fix?

GET /product/42               900ms
├─ product-service            880ms
│  ├─ SELECT product           20ms
│  ├─ SELECT review (id=1)     18ms
│  ├─ SELECT review (id=2)     19ms
│  ├─ SELECT review (id=3)     18ms
│  └─ ... 40 more SELECT review spans

🧠 Knowledge Check

Knowledge Check

In a trace waterfall, what is the 'critical path'?

Knowledge Check

You see 40 nearly identical short DB spans in a loop within one trace. What does this usually indicate?

💼 Interview Preparation

Interview Q&A

Walk me through using a trace to debug a latency spike.

Summary

You’ve completed the Zipkin track: what tracing solves, the span/trace data model and propagation, instrumenting with OpenTelemetry, and analyzing traces to debug real latency and error problems. You can now see inside a distributed request. Next, we explore MCP — the Model Context Protocol connecting AI to your tools.

Up Next

Introduction to the Model Context Protocol (MCP)

You've mastered this lesson. Continue your journey to becoming a DevOps Engineer.

Start Next Lesson→