# How to choose the layer that instruments outbound API calls > Capture one retried call at four layers, from OpenTelemetry to the gateway log, and give the span to the one that names the operation, tenant, and attempt. Source: https://voxgig.com/howto/choose-a-layer-to-instrument-outbound-calls - Audience: platform-team - Level: intermediate - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Make one call that fails and is retried, and capture it with OpenTelemetry's HTTP instrumentation, the SDK's own telemetry, the egress proxy's access log, and the gateway's log. Only an in-process layer can name the operation and the tenant. Give the span one in-process owner, number attempts with http.request.resend_count, and keep the proxy and gateway logs for the processes you cannot rebuild. --- ## You will need Node 22 or later, an SDK generated by sdkgen, and somewhere to send spans. Verified 2026-10-15 against Node 22.22.0, `@voxgig/create-sdkgen` 0.30.7, `@voxgig/sdkgen` 4.34.7, and the OpenTelemetry JS packages listed in the page's versions. The SDK under `sdk/` was generated once by `generate.sh` with the telemetry, metrics, and retry features, and is checked in as compiled JavaScript. `@voxgig/sdkgen` 4.34.7 declares Node 24 or later in its `engines` field, and `generate.sh` ran under Node 22.22.0, where npm warned and the build passed. Envoy and Kong did not run: the proxy and the gateway are Node servers on 127.0.0.1 that write a subset of the fields each product logs by default. If you have not instrumented an HTTP client with OpenTelemetry before, do that first. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [An Envoy egress proxy](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage) | Calls from processes you cannot rebuild, and one log for every outbound request | Bytes on the wire only: no operation, no tenant, and a fresh request id per attempt | You need the tenant or the operation behind a slow call | | [Kong gateway analytics](https://developer.konghq.com/plugins/http-log/) | Egress already routes through Kong and you want counts and latency per service | A gateway hop on every call, and a consumer field only when each tenant authenticates separately | Calls skip the gateway, or tenants share one credential | | [OpenTelemetry auto-instrumentation](https://opentelemetry.io/docs/languages/js/libraries/) | Services you deploy yourself, with a collector already running | Attempts named only `GET`, with no operation until your code wraps the call in a parent span | Every team uses a generated SDK that already names its operations | | [The sdkgen telemetry and metrics features](https://voxgig.com/sdk) | Clients generated by sdkgen in several languages, with the operation named for you | Default ids that W3C parsers ignore, metrics with no exporter, and counters and attempts kept on undocumented `_metrics` and `_retry` properties | A process already runs OpenTelemetry, where a second span owner splits every call | Voxgig maintains sdkgen. This page compares its telemetry and metrics features with OpenTelemetry auto-instrumentation, an Envoy egress proxy, and Kong's gateway logs. In sdkgen 4.34.7 the telemetry [feature model](https://github.com/voxgig/sdkgen/blob/v4.34.7/ts/project/.sdk/model/feature/telemetry.aontu) describes W3C trace-context propagation, but its default ids fail the W3C format. The [metrics feature](https://github.com/voxgig/sdkgen/blob/v4.34.7/ts/project/.sdk/model/feature/metrics.aontu) takes no exporter, and the generated README documents neither feature's options beyond `active`. The trade-off is fidelity against coverage. OpenTelemetry and the SDK features run inside the process, so they know the operation and the tenant, but only for processes you rebuilt. Envoy and Kong see every process on the route, including an agent you cannot change, but only as bytes. A gateway also costs a hop per call, and a proxy cannot carry its own request id forward.[^1] ## Capture one retried call at every layer The call path is SDK, egress proxy, gateway, vendor, and the vendor answers 503 once. Node stand-ins from `stands.mjs` play the proxy and the gateway, since Envoy and Kong did not run. The SDK's features are switched on in its constructor: ```js title="layers.mjs" export function meterco({ base, telemetry = true, idgen } = {}) { const sdkSpans = [] const client = new MetercoSDK({ apikey: 'sk_test_meterco', base, feature: { retry: { active: true, jitter: false }, metrics: { active: true }, telemetry: { active: telemetry, ...(idgen ? { idgen } : {}), exporter: (span) => sdkSpans.push({ ...span, tenant: callContext.getStore()?.tenant }), }, }, }) return { client, sdkSpans } } ``` The tenant comes from an [`AsyncLocalStorage`](https://nodejs.org/api/async_context.html) that your code sets. A span processor copies it onto the OpenTelemetry spans the same way. Both are code you write, and both are only possible because the layer shares the caller's process. ```bash node capture.mjs ``` ```text output one call: Meter().list() for tenant acme, vendor answers 503 then 200 OpenTelemetry undici, 2 records {"name":"GET","status":503,"error":true,"path":"/meters","tenant":"acme"} {"name":"GET","status":200,"error":false,"path":"/meters","tenant":"acme"} sdkgen features, 3 records {"span":"meter.list","ok":true,"tenant":"acme"} {"metrics":"meter.list","count":1,"ok":1,"err":0} {"retry":true,"attempt":1,"status":503,"wait":50} egress proxy (Envoy fields), 2 records "GET /meters HTTP/1.1" 503 - "Mozilla/5.0 (compatible; MetercoSDK/1.0)" "f403dc9c-9363-43db-a2d5-d722aaefcfc5" "127.0.0.1:46645" "GET /meters HTTP/1.1" 200 - "Mozilla/5.0 (compatible; MetercoSDK/1.0)" "2b45d93c-3d09-4f63-ace7-838644c13d6d" "127.0.0.1:46645" gateway (Kong http-log fields), 2 records {"request":{"method":"GET","uri":"/meters"},"response":{"status":503},"service":{"name":"meterco"},"tries":1,"consumer":null} {"request":{"method":"GET","uri":"/meters"},"response":{"status":200},"service":{"name":"meterco"},"tries":1,"consumer":null} can the layer name operation tenant attempt OpenTelemetry undici no yes no sdkgen features yes yes yes egress proxy (Envoy fields) no no no gateway (Kong http-log fields) no no no ``` The matrix is computed from the records, not written by hand. The SDK names the operation because it knows it called `list` on `Meter`. Its attempt number comes from the retry feature's record on `client._retry`, as undocumented as `_metrics`. The span is one per operation and reports the call as successful, with no trace of the 503. OpenTelemetry sees two unrelated `GET` requests. The proxy and the gateway see two requests with two request ids. Kong's `tries` counts the gateway's own balancer attempts, so it reads 1 on both.[^2] ## Give the span one owner Switch both in-process layers on and the call splits. The pitfall script runs the same retried call four ways and reports what reached the vendor: ```bash node pitfall.mjs ``` ```text output who writes traceparent spans traces headers vendor finds a parent sdkgen telemetry and OTel undici 3 3 2, 2 no sdkgen telemetry, default ids 1 1 1, 1 no sdkgen telemetry, idgen with hex ids 1 1 1, 1 yes OTel undici under an operation span 3 1 1, 1 yes traceparent as the vendor's server joined it, both layers on: 00-t0001000000000000-s0002000000000000-01, 00-<32 hex>-<16 hex>-01 traceparent from sdkgen's default ids: 00-t0001000000000000-s0002000000000000-01 ``` With both layers on, each request carries two `traceparent` headers, so a panel counting client spans counts this one call three times. The vendor's W3C parser finds no parent, because the joined value is not one header. The SDK's default ids fail on their own as well. [W3C Trace Context](https://www.w3.org/TR/trace-context/) requires a trace id of 32 lowercase hex characters, so a parser ignores `t0001000000000000`, which has 16 characters and a `t`.[^3] The span id `s0002000000000000` fails the parent id rule on its `s`. Pass an `idgen` that returns hex. The fourth row is the rule written as code. Your own span owns the operation, and every HTTP attempt is a child of it: ```js title="owner.mjs" export function operation(name, tenantId, fn) { const call = { tenant: tenantId, operation: name, sends: new Map() } return callContext.run(call, () => tracer.startActiveSpan(name, { attributes: { 'tenant.id': tenantId } }, async (span) => { try { return await fn() } catch (err) { span.recordException(err) span.setStatus({ code: SpanStatusCode.ERROR }) throw err } finally { let requests = 0 for (const n of call.sends.values()) requests += n span.setAttribute('app.http_requests', requests) span.end() } })) } ``` A request hook on the undici instrumentation numbers each repeated send per method and URL, as [`http.request.resend_count`](https://opentelemetry.io/docs/specs/semconv/http/http-spans/), which the semantic conventions ask for on every repeated span. The first send carries no attribute. ## Check it worked The suite proves the wire layers cannot name the operation or the tenant, and that two owners give two headers and three traces. It checks that a call with no retry still yields the SDK's records, since `client._retry` does not exist until the first retry. It also checks that one owner gives one trace with numbered attempts: ```js title="layers.test.mjs" assert.equal(op.attributes['tenant.id'], 'acme') assert.equal(op.attributes['app.http_requests'], 2) assert.deepEqual(attempts.map((s) => s.attributes['http.response.status_code']), [503, 200]) assert.deepEqual(attempts.map((s) => s.attributes['http.request.resend_count']), [undefined, 1]) assert.equal(new Set(spans.map((s) => s.spanContext().traceId)).size, 1, 'one call, one trace') assert.deepEqual(api.state.requests.map(traceparents), [1, 1]) ``` ```bash node --test layers.test.mjs ``` ```text output # tests 6 # suites 0 # pass 6 # fail 0 # cancelled 0 # skipped 0 # todo 0 ``` ## When it goes wrong Span counts double after a team enables SDK telemetry. Two layers write `traceparent`. Pick one: leave `feature.telemetry` off, or switch the undici instrumentation off for that process. A different request in the same operation shows up as a retry. The counter keys on the operation alone. Key it on method and URL, as the request hook here does, and a second test pins it. The metrics never reach your backend. In sdkgen 4.34.7 the metrics feature has no exporter option: it keeps counters on `client._metrics`, which the generated README does not document. Read them on a timer and export them yourself, and pin the SDK version, since an underscore property can move. ## When not to do this Do not switch on the sdkgen telemetry feature in a process that already runs the OpenTelemetry undici instrumentation. Each attempt then sends two `traceparent` headers, and a call retried once starts three traces, so the vendor's side cannot attach to any of them. Do not send sdkgen's default trace ids across a process boundary. Give the feature an `idgen` that returns hex, or leave the span to OpenTelemetry. Do not attribute cost per tenant from the gateway unless each tenant authenticates as its own Kong consumer. With one shared credential, the gateway knows the service but not the caller. Do not drop the proxy log because the in-process spans are richer. It is the only record of calls from processes you cannot rebuild, and the place to count requests the SDK never saw. ## Related how-tos - [Export outbound call spans from a serverless function](/howto/flush-spans-from-a-serverless-function) - [Tag SDK calls with session and request ids](/howto/tag-sdk-calls-with-session-and-request-ids) - [Route outbound API calls through an egress proxy](/howto/route-outbound-calls-through-an-egress-proxy) ## Last verified Verified 2026-10-15 against Node 22.22.0, `@voxgig/create-sdkgen` 0.30.7, `@voxgig/sdkgen` 4.34.7, `@opentelemetry/sdk-trace-node` 2.11.0 and `@opentelemetry/instrumentation-undici` 0.32.0. Every output block is what the command preceding it printed. The proxy and the gateway are loopback stand-ins that record a subset of Envoy's and Kong's default fields, in place of the products themselves. [^1]: Envoy's [header documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers) says Envoy generates an `x-request-id` for requests that lack one, then adds that the header cannot be forwarded automatically, because of Envoy's out of process architecture. It calls this one of the few areas where a thin client library is needed. The proxy that sees every request needs the client's help to connect one request to the next. [^2]: Kong's [log format](https://developer.konghq.com/plugins/http-log/) defines `tries` as the iterations the load balancer made for this request. A retry by the client arrives as a separate request with a `tries` list of its own, so the gateway's retry count describes the gateway alone. [^3]: The [specification](https://www.w3.org/TR/trace-context/) gives the trace id 32 lowercase hex characters, forbids the value that is all zeros, and tells vendors to ignore a `traceparent` whose trace id contains anything else. The sdkgen [feature model](https://github.com/voxgig/sdkgen/blob/v4.34.7/ts/project/.sdk/model/feature/telemetry.aontu) describes itself as distributed tracing with W3C trace-context propagation. Its [default ids](https://github.com/voxgig/sdkgen/blob/v4.34.7/ts/project/.sdk/tm/ts/src/feature/telemetry/TelemetryFeature.ts) are sequential, unique within one client, and readable at a glance, three properties the specification did not ask for.