How to Trace Code Latency Spikes in Production

A latency chart tells you when an application became slow. It does not tell you which request, dependency, or code path caused the delay.

The best tools to trace code latency spikes combine four views:

1. Metrics show the affected time window and route.
2. Distributed traces show where a slow request spent its time.
3. Custom spans expose application work that automatic instrumentation misses.
4. A profiler explains CPU, allocation, or lock time inside one broad span.

This guide gives you a repeatable production workflow. It also shows when each tool answers the next debugging question.

Quick answer: how do you trace a latency spike?

Start with P95 or P99 latency for one service and route. Choose a slow request from the same time window. Open its trace waterfall and follow the longest causal path.

Inspect database, client, messaging, and internal spans before you change code. Add a custom span where the trace contains an unexplained gap. Use a profiler only when the remaining delay stays inside application code.

After the fix, compare the same route, percentile, traffic class, and release. A lower average can hide a worse tail.

Use the tool that answers your current question

No single latency tool explains every delay. Each signal has a different job.

OpenTelemetry defines traces as request paths and metrics as runtime measurements. Profiles record resource use at code level. Use these signals together instead of forcing one signal to answer every question.

Step 1: define the latency symptom

Write a narrow statement before you inspect a trace.

P95 latency for POST /checkout increased from the normal range after the latest release. Error rate stayed flat.

Record these dimensions:

• Service and operation
• Time window
• P50, P95, and P99 latency
• Request rate and error rate
• Region, tenant, or traffic class when relevant
• Current and previous release

Google's four golden signals include latency, traffic, errors, and saturation. Check them together. A latency spike during a traffic surge needs different evidence from a spike after a release.

Do not begin with the average. Ten very slow requests can disappear inside thousands of fast requests. P95 shows the threshold that 95 percent of observed requests complete below.

Step 2: select a representative slow trace

Choose a trace from the affected route and time window. Prefer a trace near the slow percentile you are investigating.

Check the root span first:

• Does its route match the affected operation?
• Does its duration match the latency chart?
• Does it include the expected service and release attributes?
• Does it have complete parent-child relationships?
• Did sampling retain the slow request and its children?

One trace is an example, not a trend. Inspect several slow traces before you name the cause. Compare them with a normal trace from the same route.

Step 3: follow the critical path

A trace waterfall shows spans against time. The widest visible span is not always the cause.

Follow the causal path from the root span. Look for child spans that end near their parent's end time. These spans often control the user-visible duration.

Pay attention to overlap. Two parallel 400 ms calls do not add 800 ms to the request. Two sequential 400 ms calls can.

Use this example:

POST /checkout                         1,840 ms
├─ validate cart                          24 ms
├─ reserve inventory                     612 ms
│  └─ POST inventory-service             594 ms
├─ charge payment                        948 ms
│  └─ POST payment-provider              921 ms
└─ save order                            188 ms
   └─ INSERT orders                      171 ms

The payment call dominates this request. The database is slower than normal, but it is not the first investigation target.

OpenTelemetry spans include start and end timestamps, attributes, events, status, and parent context. The OpenTelemetry tracing API explains this structure.

Step 4: classify the delay

Most code latency spikes fit one of five evidence patterns.

Database delay

Look for repeated database spans, long query spans, or time between query operations.

Repeated similar spans can indicate an N+1 query pattern. A single long span can indicate a missing index, lock wait, or large result set. The trace identifies the operation. Database query plans and database metrics confirm the cause.

Use the N+1 query regression guide when the waterfall shows repeated database work.

Downstream service delay

Look for a long client span with a matching server span in the downstream service.

If the client span is long but the server span is short, inspect network time, retries, connection pools, and queues. If the downstream server span is also long, continue the trace inside that service.

W3C Trace Context standardizes traceparent and tracestate propagation. Broken propagation can split one request into separate traces.

Application code delay

A broad application span with no detailed children creates an instrumentation gap. Add spans around meaningful stages, not every function.

Useful span boundaries include:

• Serialization of a large response
• A cache fallback
• A pricing calculation
• A queue publish or consume stage
• A retry loop
• A lock-protected section

Name spans by stable operations. Add low-cardinality attributes that help comparison. Avoid user IDs, raw payloads, and secrets.

If the added spans still leave a broad CPU-bound section, use a profiler. Tracekit does not provide continuous profiling. A profiler is the correct next tool for function-level CPU, allocation, or lock evidence.

Queue or pool delay

Request work can wait before a database call or network call begins. Inspect connection pools, worker queues, thread pools, and retry backoff.

Measure queue time separately from execution time. A fast dependency can still produce slow requests when every caller waits for a scarce connection.

Network delay

Measure DNS, connection, TLS, and time to first byte separately. A distributed trace can show that time sits outside application spans. Network tools confirm the transport cause.

Do not use ping alone to explain API latency. Ping measures a different protocol and does not include application processing.

Step 5: close instrumentation gaps safely

Automatic instrumentation usually covers HTTP, database, and common client libraries. Your business logic can still remain invisible.

Add a custom span when you need duration and trace relationships. Add a Tracekit dynamic log when duration is known but runtime state is missing.

For example, a span can prove that calculate-discount took 480 ms. A conditioned capture point can then record bounded runtime state for only the affected path.

Tracekit dynamic logs capture variables, stack traces, and trace context from running traffic. They do not pause the application. They are bounded capture points, not general log ingestion.

Use these safety rules:

• Capture only the values needed for one hypothesis.
• Apply a condition or sampling limit.
• Keep PII scrubbing enabled.
• Set a maximum capture count.
• Disable the capture point after the investigation.

See the dynamic logs documentation for supported SDKs and safety controls.

A Tracekit latency investigation

Tracekit accepts standard OTLP traces and metrics. You can keep existing OpenTelemetry instrumentation and change the exporter endpoint.

A practical workflow is:

1. Use service metrics to find the affected P95 or P99 window.
2. Filter traces by service, operation, duration, and time.
3. Open a slow trace and inspect its span waterfall.
4. Use the service map when latency crosses service boundaries.
5. Add custom spans for missing application stages.
6. Add a bounded dynamic log when runtime state explains the slow path.
7. Compare the same operation after the fix.

Tracekit stores trace duration and supports minimum and maximum duration filters. Its service views calculate P50, P95, and P99 latency. Latency alert rules support service or endpoint scope.

You can also use the Tracekit MCP server to ask for slow requests and trace details. The MCP tools are read-only. Always inspect the returned trace evidence before you change code.

If you already export OpenTelemetry data, use the OTLP integration guide. The setup does not require a Tracekit SDK for traces and metrics.

Validate the fix against the same slice

Do not close the incident after one fast request.

Compare before and after data for:

• The same service and route
• The same P95 or P99 percentile
• A similar traffic window
• The same region or tenant class
• Error rate and throughput
• The new release identifier

Then compare trace shape. Confirm that the target span became shorter and no new sequential work appeared.

Set a latency alert after you understand the normal baseline. Tracekit supports average, P50, P95, and P99 latency conditions. The alert rules guide explains scopes, windows, and cooldowns.

For baseline and alerting options, compare these anomaly detection tools for small development teams.

Latency investigation checklist

• Define the affected service, route, window, and percentile.
• Check traffic, errors, saturation, and release timing.
• Select several slow traces from the same slice.
• Compare each slow trace with a normal trace.
• Follow the critical path and account for parallel spans.
• Check database, downstream, queue, code, and network evidence.
• Add custom spans only where the trace has gaps.
• Use dynamic logs only when runtime state tests a clear hypothesis.
• Use a profiler for unresolved function-level CPU or lock time.
• Validate the fix against the same percentile and traffic class.

Frequently asked questions

Which tool is best for tracing code latency spikes?

Use distributed tracing first when the slow action crosses services or dependencies. Use metrics to choose the affected window. Use a profiler after the trace isolates the delay inside one application span.

Can traces find a slow function?

Traces can find a slow function when instrumentation creates a span for that function or stage. Otherwise, add a meaningful custom span. Use a profiler when function-level CPU evidence is required.

Why does P99 increase while average latency stays stable?

A small group of very slow requests can move P99 without changing the average much. Filter those requests by route, service, release, and traffic class.

Do dynamic logs replace distributed tracing?

No. Traces show request structure and time. Dynamic logs capture bounded runtime state at a selected code location. Use the trace to decide where state is useful.

Should I add a span around every function?

No. Instrument stable operations and important boundaries. Too many spans add cost and make the waterfall harder to read.

Trace the request before you change the code

The fastest latency workflow moves from a broad signal to specific evidence.

Use metrics to define the spike. Use traces to find the slow path. Add custom spans for missing stages. Use dynamic logs for targeted runtime state. Use a profiler when the remaining question is inside one function.

This order prevents random code changes. It also gives you a clear before-and-after test for the fix.