Latency instrumentation

Every service on the position path writes one log line per request, and the lines of one CAMARA request share a correlator. The platform joins them into the time spent at each hop. This page is the contract of that join. The stack writes the lines, and KELT collects and aggregates them.

Correlator

The CAMARA x-correlator header ties the hops together:

• Where a request enters the stack from outside, the correlator is taken from the request, or generated, and returned on the response: at the gateway, and at the wifi-adapter's POST /ingest/wifi-scan, where scans arrive from the edge. Every ingest therefore writes a line, accepted or refused.
• Every internal call forwards it: gateway to engine, engine to adapter, vendor-adapter to the vendor cloud.

Requests without a correlator write no line. The engine's position broadcast polls the adapters without one, so the stream leaves no trace.

The line

camara-gateway, positioning-engine, wifi-adapter, vendor-adapter and synthetic-adapter each write one line per request, at INFO on the hop logger, as one JSON object. The schema is schema/hop-log.schema.json, also served by the gateway at GET /contracts/hop-log.schema.json. A request that fails with an unhandled error, or is aborted before it answers, still writes its line, with status 500.

{
  "event": "hop",
  "service": "camara-gateway",
  "stage": "POST /location-retrieval/v0.5/retrieve",
  "correlator": "b6f1c2d4-7a0e-4a3b-9e61-2f1f3c7d8a90",
  "status": 200,
  "t_receive": 1790000000.123456,
  "t_emit": 1790000000.145678,
  "span_ms": 22.222
}

Field	Meaning
event	always hop, to select these lines in a mixed log
service	the service that wrote the line
stage	method and path of the request served
correlator	the x-correlator, the join key
status	the HTTP status returned
t_receive, t_emit	epoch seconds when the request arrived and when the response left
span_ms	t_emit - t_receive in milliseconds, including the time spent waiting on downstream hops
cache	on the vendor-adapter's GET /measurement/{id} only: hit when it answered from its cache, miss when it called the vendor
connection	with cache: miss: new when the call opened a connection (TCP and TLS setup inside the span), reused when it used a kept one
upstream_ms	with cache: miss: milliseconds from the request sent to the vendor until its response headers arrived, the vendor's own time plus the round trip

Aggregation

Group the lines by correlator and order them by t_receive:

• End to end: the gateway line's span_ms.
• Own time of a hop: its span_ms minus the span_ms of the hops it called.
• Vendor cloud: the vendor writes no line, so its time is inside the vendor-adapter's own time on a line with cache: miss. A comparison without the network to the vendor uses make demo, where the vendor-adapter calls the local mock-vendor.
• Caches: the gateway reuses a position fetched less than LOCATION_CACHE_TTL_S ago, and the vendor-adapter a vendor response younger than the schema's cacheTtl. A gateway hit leaves no engine line for its correlator. A vendor-adapter hit carries cache: hit. A request with maxAge: 0 always reaches the engine. It reaches the vendor only when the vendor-adapter's cache holds no response for the device, so other consumers that poll the same device (the stream, the location-app detail panel) keep it warm.