Observability Interview Questions India 2026: Logs, Metrics, Traces, and OpenTelemetry

Observability fundamentals:

1. What is observability? Observability is the ability to understand the internal state of a system from its external outputs (logs, metrics, traces). A system is highly observable if you can ask new questions about its behaviour without modifying it. Observability vs monitoring: monitoring checks known failure modes (you define the alerts upfront). Observability allows you to investigate unknown failure modes (explore the system's state without pre-defined dashboards).

2. Logs: Logs are timestamped records of discrete events. Structured logs: JSON format (machine-parseable); preferred over unstructured text logs. Key fields: timestamp, log level (INFO, WARN, ERROR), service name, trace ID (correlates logs to a trace), message, and relevant context (user ID, order ID, request ID). Log levels: DEBUG (very verbose; development only), INFO (normal operations), WARN (unexpected but handled), ERROR (something failed; requires investigation), FATAL (application cannot continue). Log management: ELK stack (Elasticsearch, Logstash, Kibana), Loki + Promtail + Grafana (lightweight; labels instead of full indexing; cheaper), Datadog Logs (managed; expensive).

3. Metrics: Metrics are aggregated numerical measurements over time. Prometheus metric types: Counter (monotonically increasing value: total HTTP requests, total errors), Gauge (value that goes up and down: current memory usage, number of active connections), Histogram (distribution of values: request latency in 10ms, 50ms, 100ms, 500ms buckets), Summary (pre-calculated quantiles: p50, p95, p99 latency). Time series database: Prometheus scrapes metrics from application /metrics endpoints (pull model); stores time series data; provides PromQL for querying. Grafana: dashboards and alerting on top of Prometheus.

4. Traces: A trace represents the path of a single request through multiple services. Each step in the request's path is a span (contains: span ID, parent span ID, operation name, start time, duration, status, key-value tags). A complete trace is the tree of spans for one request. Distributed tracing backends: Jaeger (CNCF graduated; open-source), Zipkin, Tempo (Grafana's tracing backend), commercial (Datadog APM, Dynatrace). OpenTelemetry: vendor-neutral instrumentation library (generate traces, metrics, and logs in a standard format; export to any backend).

Observability instrumentation:

1. What is OpenTelemetry? OpenTelemetry (OTel) is a CNCF project that provides: standardised APIs and SDKs for generating telemetry data (traces, metrics, logs) in multiple languages (Java, Go, Python, Node.js, .NET), a collector (a standalone component that receives, processes, and exports telemetry data to various backends), and the OpenTelemetry Protocol (OTLP) for transmitting telemetry data. Benefit: instrument your application once with OpenTelemetry; send the data to any backend (Jaeger, Prometheus, Datadog, AWS X-Ray) without changing the instrumentation code. Avoids vendor lock-in.

2. Automatic vs manual instrumentation: Automatic instrumentation: OpenTelemetry agents automatically instrument popular libraries and frameworks (Spring Boot, Express, Django, gRPC, HTTP clients) without code changes. Generates traces for all incoming and outgoing HTTP calls, database queries, and message queue operations. Manual instrumentation: add custom spans and attributes for business-level context (add the orderid and userid to spans so you can search for traces by order). Use when automatic instrumentation does not capture the relevant business context.

3. Context propagation: For distributed tracing to work across services, trace context (trace ID, span ID) must be propagated with every request. The W3C Trace Context standard defines the traceparent HTTP header format. OpenTelemetry SDKs automatically inject the traceparent header into outgoing HTTP calls and extract it from incoming calls. Without context propagation, each service starts a new trace and the end-to-end request path cannot be reconstructed.

4. The OpenTelemetry Collector: The OTel Collector is a middleware component deployed alongside your application. It receives telemetry data from the application (via OTLP or legacy protocols), processes it (batch, sample, filter, transform), and exports it to one or more backends (send traces to Jaeger, metrics to Prometheus, logs to Loki). Using the Collector: decouples the application from the backend (change backends without modifying the application), adds processing (tail-based sampling: only export traces for slow or failed requests), and reduces the number of connections from each application instance.

Metrics and alerting:

1. Prometheus architecture: Prometheus uses a pull model: it scrapes /metrics endpoints on a configurable interval. ServiceMonitor (Kubernetes): the Prometheus Operator uses ServiceMonitor custom resources to discover which services to scrape (based on labels). PromQL: Prometheus's query language for querying and aggregating time series data: ```promql # Error rate (errors per second over 5 minutes) rate(httprequeststotal{status=~'5..'}[5m])

# p99 latency histogramquantile(0.99, rate(httprequestdurationseconds_bucket[5m]))

# Memory usage as a percentage of the limit containermemoryworkingsetbytes / containerspecmemorylimitbytes * 100 ```

2. Alerting: Prometheus Alertmanager: receives alerts from Prometheus (when a PromQL expression is true for a specified duration), deduplicates and groups alerts, and routes them to notification channels (PagerDuty, Slack, email). Alert examples: `errorrate > 0.01 for: 5m` (more than 1% errors for 5 minutes), `cpuusage > 80 for: 10m` (CPU above 80% for 10 minutes). Good alerts: alert on symptoms (user-facing impact), not causes (low-level metrics that may not affect users). Silence alerts during maintenance windows.

3. The four golden signals: Originating from Google's SRE book: Latency: how long it takes to serve a request (monitor both successful and error latency separately; a fast error is not the same as a fast success). Traffic: how much demand the system is handling (requests per second, messages per second). Errors: rate of failed requests (HTTP 5xx, timeouts, wrong results). Saturation: how full the system is (the resource that is most constrained: CPU, memory, disk, connections).

4. SLOs and error budgets: SLO (Service Level Objective): a target value for an SLI. Example: 99.9% of HTTP requests succeed in under 500ms over a 30-day window. Error budget: the allowable amount of unreliability (1 - SLO). For 99.9% SLO: 0.1% error budget = 43.8 minutes of downtime per month. Error budget policy: if the error budget is exhausted, stop shipping new features and focus on reliability. Burn rate alerts: alert when the error budget is being consumed faster than expected (burning at 14x the normal rate for 1 hour means the monthly budget will be exhausted in 2 days).

Tracing and incidents:

1. Using distributed tracing to debug performance: Scenario: users report that the checkout page is slow. Steps: (1) Find traces for the checkout endpoint with high latency (search Jaeger for traces where the root span duration > 3s). (2) Examine the span breakdown: which child service took the most time? (3) Identify the slow span: the inventory service takes 2.5s. (4) Look at the inventory service's database spans: a query is taking 2s. (5) Fix: add a missing database index. Without distributed tracing, debugging this would require checking each service's logs separately and trying to correlate timestamps manually.

2. Log correlation with traces: For maximum debuggability, include the trace ID in every log line. When an error occurs, you can: find the error in the log aggregation tool, copy the trace ID, and look up the full distributed trace to see the end-to-end request context. OpenTelemetry SDKs automatically inject the current trace ID into log statements via MDC (Mapped Diagnostic Context in Java) or similar mechanisms.

3. Incident management: Detection: PagerDuty alert fires (error rate SLO burn rate alert). Triage: incident commander declares SEV1; starts an incident Slack channel; pulls in the on-call engineers for the affected services. Mitigation: identify the mitigation (rollback the deployment from 2 hours ago), execute it, confirm recovery (error rate back to baseline). Communication: status page update every 15 minutes during SEV1. Root cause analysis: after mitigation, investigate why (using logs, metrics, and traces). Post-mortem: blameless; 5 Whys root cause analysis; action items with owners and due dates.

4. On-call best practices: Runbooks: documented procedures for common alerts (what to check, how to mitigate, when to escalate). On-call rotation: weekly or bi-weekly; engineers rotate through on-call duty; always a primary and a secondary (backup). Toil reduction: SRE principle; if an alert fires more than twice, automate the mitigation; if a task is repetitive and manual, automate it. Post-on-call review: after each on-call week, review alerts that fired (are they actionable? Should they be modified or removed?).