Three signals, four stores, one place to look.

Prometheus or ClickHouse for metrics. Loki or ClickHouse for logs. Tempo or ClickHouse for traces. Dashboards mix them, the agent queries all of them, and a log line with a trace id opens the trace.

PrometheusClickHouseLokiTempo

observe.example.com/logs?ds=loki&q={namespace="shop"} | json

LogsLast 15 minutesLive tailAdd to dashboard

DatasourceLoki (Loki)Label browser

1{namespace="shop"} | json

Run query

Order Newest firstWrap linesShow timeDeduplication No deduplication

2026-11-06 14:02:11.204app=checkout pod=checkout-0Errorpayment declined order=8812 reason=card_declined
2026-11-06 14:02:11.198app=frontend pod=frontend-1InfoGET /cart 200 12ms
2026-11-06 14:02:10.911app=checkout pod=checkout-0Warningdb pool near max in_use=19 max=20
2026-11-06 14:02:10.877app=payments pod=payments-0Infocharge ok amount=4900
2026-11-06 14:02:09.540app=checkout pod=checkout-0Errorupstream timeout upstream=payments after=2.0s
2026-11-06 14:02:09.102app=frontend pod=frontend-0InfoGET /product/41 200 8ms
2026-11-06 14:02:08.990app=inventory pod=inventory-0Inforeserve ok sku=41 qty=1
2026-11-06 14:02:08.421app=checkout pod=checkout-0Errorpayment declined order=8809 reason=timeout

Show contextOpen trace

Time2026-11-06 14:02:09Bodyupstream timeout upstream=payments after=2.0sappcheckoutdetected_levelerrornamespaceshoppodcheckout-0trace_id7f3a9c1e5b2d4f6a8c0e1d3b5a7f9c2e

observe.example.com/t/7f3a9c1e5b2d4f6a8c0e1d3b5a7f9c2e?ds=tempo

Trace 7f3a9c1e7f3a9c1e5b2d4f6a8c0e1d3b5a7f9c2e

2026-11-06 13:02:09 to 2026-11-06 15:02:09Copy linkSearch traces

frontend POST /checkoutTrace 7f3a9c1e5b2d4f6a8c0e1d3b5a7f9c2e2026-11-06 14:02:09.1122.31 s

Service and operation

frontend POST /checkoutError2.31 s

checkout place_orderError1.98 s

inventory reserve210 ms

checkout db.query90 ms

payments POST /chargeError1.52 s

payments authorize_cardError1.49 s

frontend render110 ms

payments POST /chargeSpan 9f2c1a7b3e4d5f602026-11-06 14:02:09.6631.52 sError

http.request.methodPOSThttp.response.status_code504server.addressgateway.psp.examplek8s.pod.namepayments-0service.namepayments

The trace view.

A waterfall per trace, the slow span selected, its attributes beside it. Tempo or the OpenTelemetry tables in ClickHouse, with TraceQL or SQL behind it.

Try a query before you save it.

The logs page takes LogQL on Loki or SQL on ClickHouse. "Show context" brings up the lines around a line, a line with a trace id opens its trace, and "Add to dashboard" saves the query as a logs panel. Trace search finds traces by service, span, duration or TraceQL. In the panel editor, the preview runs PromQL or SQL on the real datasource before you save.

observe.example.com/d/4f1c8a2e-93b7-4d1a-b6e5-0c2f7d9a1e34/checkout-health?panel=latency_by_route&edit=query

Edit p95 latency by routeA draft. Nothing changes until you save.

Back to activityEditorJSON

Draft ran just now, 3 seriesLast 6 hours

Ask the agent to change this query, like: only 5xx, by routeSuggest

Datasourceprom (Prometheus)Metrics and labels

Query APanel datasource (prom)

1histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket{job="checkout"}[$__rate_interval])))

Query is valid. Errors show under the line that has them.ExemplarsLegend{{route}}+ Add query

Panel typeTime series

Unitseconds (s)

ThresholdsAdd threshold

Titlep95 latency by route

1 change: queryAdd a noteDiscardSave as version 8

What each store gives the agent.

Store	Metrics	Logs	Traces	Discovery
Prometheus	PromQL	none	none	metric inventory by family, labels, label values, series per family
ClickHouse	SQL	SQL	SQL	tables in the configured database, columns and types, sample rows
Loki	none	LogQL	none	stream labels and values, line samples
Tempo	none	none	TraceQL	service names, span names, attribute keys and values

Every query is time-bounded and limited on the server, whatever the prompt says. Credentials never reach the browser.