Metrics & Tracing9 min read•September 2026•Kernelry Staff

Observability, OpenTelemetry & Metrics Interview Scenarios

Modern cloud architectures have shifted from fragmented monitoring tools toward unified observability powered by OpenTelemetry. Engineering reliable systems requires diagnosing distributed latency bottlenecks across microservices, mitigating high-cardinality time-series explosions, and building actionable SLO alerting.

Architecture Blueprint

OpenTelemetry Telemetry Ingestion & Unified Pipeline

Visualizing how microservices send traces and metrics to the OTel Collector, which batches, sanitizes, and exports to Prometheus, Loki, and Tempo.

Architecture Blueprint: OpenTelemetry Telemetry PipelineObservability

The Core Concept

Observability goes beyond basic dashboards. It is the ability to understand how a distributed system is behaving internally by inspecting its telemetry. Modern architectures combine metrics for alerting, structured logs for context, and distributed traces for request latency into a unified OpenTelemetry pipeline.

Key Rules of Thumb for Production Systems

•The Four Golden Signals of SRE are Latency, Traffic, Errors, and Saturation. Prefer user-impact signals, while allowing a causal signal to page when it reliably predicts imminent harm and has a documented action.

•High cardinality occurs when metric labels contain unbounded unique values (like user IDs or UUIDs), crashing Prometheus TSDBs.

•Distributed tracing propagates TraceParent headers across HTTP/gRPC boundaries to reconstruct request execution timelines.

•Use tail-based sampling policies that retain enough error evidence for investigation while accounting for volume, cost, privacy, and sensitive-attribute redaction.

Why Production Systems Demand This Depth

When an API reports 504 gateway timeouts across 20 interconnected microservices, CPU graphs alone will not reveal the issue. Senior engineers use distributed tracing spans, trace context propagation, and error rates to pinpoint the failing service in minutes.

Common Diagnostic Pitfalls

•Putting high-cardinality labels (like User IDs, IP addresses, or timestamps) inside Prometheus metric labels, causing memory explosion.
•Alerting on noisy static thresholds (e.g. CPU > 85%) instead of user-impacting symptoms (SLO error budget burn rate).
•Collecting billions of log lines without structured JSON format, making automated searching and parsing slow and costly.
•Failing to propagate W3C TraceContext headers across internal HTTP/gRPC client calls, breaking distributed trace visibility.

Best Practices for Scenario Drills

•Reference the Google SRE Four Golden Signals (Latency, Traffic, Errors, Saturation) and the USE/RED methods.
•Explain why OpenTelemetry has become the vendor-neutral standard for instrumentation (write once, export to Datadog, Prometheus, or Honeycomb).
•Discuss head-based sampling vs tail-based sampling in distributed tracing.

Worked Production Scenarios

Study real-world outage patterns, diagnostic trees, and clarifying questions.

Mitigating High-Cardinality Telemetry Explosions in Ingestion Pipelines

Difficulty: mid

Scenario Prompt:

“After a new microservice release, your Prometheus TSDB memory consumption triples in 20 minutes, causing OOM crashes. You discover a developer added user_id="<uuid>" as a label to the http_requests_total metric. How do you resolve this immediately and prevent recurrences?”

Substantive Technical Concepts to Hit:

Understanding Metric Cardinality

Every unique combination of label key-value pairs creates a new time-series in memory. Adding millions of unique user UUIDs produces millions of parallel time-series, consuming gigabytes of RAM.

Metric Relabeling & Dropping

Apply a metric_relabel_configs rule in Prometheus or an OTel Collector processor to strip the user_id label before data touches the TSDB.

Logging vs Metrics Separation

Educate developers that high-cardinality identifiers belong in structured logs or distributed trace span attributes, not in time-series metric labels.

Automated CI Linting for Telemetry

Implement pre-merge linting rules that reject metric definitions containing known unbounded identifier names.

Senior Clarifying Questions to Investigate:

Can we immediately deploy a Prometheus relabeling config reload without taking down the monitoring stack?
Is an OpenTelemetry Collector deployed between the application pods and the Prometheus server to filter the label upstream?
Do we need to retain the per-user data in our log aggregator (Loki/OpenSearch) for billing or fraud analysis?

Debugging Tail Latency Spikes Using Distributed Traces

Difficulty: senior

Scenario Prompt:

“A checkout API has a P50 latency of 35ms, but the P99 latency spikes to 4,200ms. CPU, memory, and database connection pools appear completely normal. How do you use distributed tracing to find the hidden bottleneck?”

Substantive Technical Concepts to Hit:

Trace Span Waterfall Analysis

Filter traces in Jaeger/Tempo specifically for duration > 4000ms. Inspect the waterfall timeline to identify which specific child span accounts for the unaccounted time.

Downstream Network / Lock Contention

Identify whether the delay is caused by an external third-party payment gateway timeout, thread pool starvation, or distributed lock acquisition.

Context Propagation Verification

Verify that W3C traceparent headers are properly forwarded across asynchronous message queues (Kafka/RabbitMQ) to maintain trace continuity.

Tail-Based Sampling Optimization

Configure OTel Collector tail-based sampling to always capture 100% of traces exceeding the P99 latency threshold for root cause analysis.

Senior Clarifying Questions to Investigate:

Are all downstream microservices instrumented with OpenTelemetry context propagation?
Does the span waterfall show time spent executing inside a service or waiting in a queue between services?
Is the tail latency correlated with specific payload sizes or tenant accounts?

Interactive Scenario Drills

That's where we come in.

Explaining architectural trade-offs under time constraints is very different from reading about them.

Kernelry helps you rehearse technical questions, evidence-led debugging, safe next steps, and the clear communication teams expect. The drills adapt to your answer, so you practice the reasoning rather than memorize a company-specific script. During early access, free accounts include ten daily scenarios across our mixed question pool, with no credit card required.

Start Today's Free Drill

Back to All Blog Posts