Observability (OpenTelemetry)

CxReports can emit OpenTelemetry traces and metrics over OTLP — the vendor-neutral protocol understood by Grafana Tempo/Mimir, Datadog, Elastic, Honeycomb, and most other observability backends. Nothing is emitted unless you configure an endpoint. With no endpoint set, the telemetry SDK is not even registered — there is no background exporter, no buffering, and no possibility of accidental egress.

This page exists so a security reviewer can decide, without reading source code, exactly what leaves the server. Where a guarantee has a limit, that limit is stated here rather than smoothed over.

When to use this

Turn this on if you run CxReports behind an existing observability stack and want request-level tracing (which report, which data source, how long PDF rendering took) and dashboards (throughput, error rate, queue depth). It's optional; CxReports runs exactly as it always has if you never set an endpoint.

Turning it on

Set these as environment variables (standard OpenTelemetry SDK variables — CxReports doesn't invent its own):

Setting	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	The OTLP collector URL. Setting this is what turns telemetry on at all.
OTEL_EXPORTER_OTLP_HEADERS	Extra headers sent with every export call — typically an API key, as key1=value1,key2=value2.
OTEL_EXPORTER_OTLP_PROTOCOL	grpc (default) or http/protobuf, depending on what your collector expects.
OTEL_SERVICE_NAME	The service.name resource attribute. Defaults to cxreports.
OTEL_TRACES_SAMPLER	Standard OTel sampler selection (e.g. always_on, parentbased_traceidratio). If you don't set it, CxReports samples at parent-based ratio 0.1 — roughly one trace in ten is exported, so report generation isn't paying to export a span per data source on every request. Setting this variable replaces that default entirely; set always_on if you want every trace. Metrics are unaffected: they are aggregates, and every recorded value is counted whether or not its trace is sampled.
Telemetry:DetailLevel	Standard (default) or Detailed — see below. In appsettings.json use Telemetry:DetailLevel; as an environment variable, Telemetry__DetailLevel.

Every instance in a multi-instance deployment should be configured identically, the same as other High Availability settings.

What is emitted

CxReports' own code emits telemetry through one central class (CxReportsTelemetry/CxReportsMetrics internally), so every span and metric name below is exhaustive for what CxReports itself produces — there is no other code path that starts a span or records a metric. On top of that, three third-party instrumentation libraries are enabled (ASP.NET Core, HttpClient/Npgsql — see Third-party instrumentation below for what that means for this guarantee).

Spans

Span name	Emitted by	Attributes	Notes
cxreports.report.generate	Report export/generation	workspace.id, report.type, output.format, outcome	Top-level span for one document generation.
cxreports.pdf.postprocess	PDF post-processing (merge/overlay/protect)	pdf.protected, pdf.merge, outcome	Child of cxreports.report.generate.
cxreports.browser.render	Headless browser page checkout for rendering	outcome	Covers waiting for a pool slot and navigating the render page; outcome is error on pool-timeout as well as on render failure.
cxreports.datasource.execute	Data source execution (SQL, API, Mongo, etc.)	source.type, outcome
cxreports.job.enqueue	Starting a workflow	workflow.name, outcome	Measures initiation only — validating the definition, persisting initial state, enqueueing the start event. It is not a "job duration" span; see below.
cxreports.job.task	A worker executing one SIMPLE workflow task	task.type, outcome	Independent trace root, not a child of cxreports.job.enqueue. See coverage gap below — only SIMPLE tasks emit this.
cxreports.delivery.upload	A report delivery (email, S3, SFTP, SharePoint, Google Drive, local volume)	delivery.type, outcome	Deliberately excludes the remote path, object key, bucket, host, or filename — a generated filename routinely contains a person's name.

Every span records an outcome of ok or error, and on error also sets the span status to error and attaches an exception event. There are no exceptions to this — filtering on outcome="error" finds every failure CxReports' own instrumentation saw.

Metrics

Metric name	Type	Attributes	Notes
cxreports.report.generation.duration	Histogram (ms)	outcome, output.format, trigger	trigger is one of interactive, scheduled, api, background — which caller produced the report. Recorded at the single export chokepoint, so every generation is counted regardless of how it was requested.
cxreports.browser.render.duration	Histogram (ms)	outcome
cxreports.browser.pool.in_use	Observable gauge	(none)	Read directly from the render-page pool's own semaphores, not a counter CxReports maintains.
cxreports.delivery.duration	Histogram (ms)	delivery.type, outcome
cxreports.datasource.execute.duration	Histogram (ms)	source.type, outcome
cxreports.job.queue.depth	Observable gauge	(none)	Read directly from Hangfire's monitoring API across all queues, not a counter CxReports maintains.

workspace.id, report.id, template.name, and user.id are never present on a metric, by construction — see Cardinality below.

Exceptions

When CxReports records an exception on a span, it always includes exception.type and exception.stacktrace (stack frames only). exception.message is included only at Telemetry:DetailLevel=Detailed, because exception messages from SQL drivers, Mongo drivers, and delivery targets routinely embed query fragments, connection details, or file paths.

One deliberate consequence: inner-exception detail is not captured even at Detailed. The stack trace is taken from Exception.StackTrace, not Exception.ToString() — ToString() prefixes the exception message onto the trace text, which would leak the message back into "stacktrace" even when exception.message itself is withheld at Standard. If you need full inner-exception detail, correlate the trace with the corresponding server log entry.

The job spans are not parent and child

cxreports.job.enqueue and cxreports.job.task look like they belong in the same trace, but they don't, and treating them that way will hide problems rather than reveal them.

CxReports' job engine is event-sourced: starting a workflow enqueues a WorkflowStartedEvent and returns — cxreports.job.enqueue ends there, typically in low single-digit milliseconds. The workflow itself executes later, asynchronously, driven by a separate event-processing loop that may run on a different instance and at an arbitrarily later time. No trace context crosses that event store, so each cxreports.job.task span starts as its own independent trace root, not a child of the enqueue span.

If you build a "job duration" dashboard panel on cxreports.job.enqueue, it will look permanently healthy while actual work backs up — that span cannot see the backlog by design. Use cxreports.job.queue.depth to watch for backed-up work, and inspect cxreports.job.task spans (grouped by task.type, correlated by time rather than by trace) for task-level execution time.

Known coverage gap

Only worker-executed SIMPLE tasks emit cxreports.job.task. Inline system tasks — Http, Switch, DoWhile, ForkJoin, SubWorkflow — are not instrumented. For a workflow that's heavy on control flow rather than SIMPLE task dispatch, most of its execution time is currently untraced. This is a known gap, not an oversight to work around; if you need visibility into a specific control-flow-heavy workflow today, correlate with application logs.

Detail levels

Level	What it changes
Standard (default)	Safe for production by default. Strips the reviewed denylist of sensitive keys — query strings, request/response bodies, HTTP headers, SQL query text, generated document filenames, exception messages — from everything CxReports exports. See Third-party instrumentation, and its limits for what "strips" does and does not cover.
Detailed	Stops stripping exception.message and db.query.text (SQL statement text), so both reach your backend. Everything else is identical to Standard. Turn this on only where the destination and access controls for your telemetry backend meet the same bar as your application logs — SQL text and exception messages can contain data values.

How SQL text is actually handled

CxReports does not configure the PostgreSQL instrumentation at all, in either direction. The Npgsql driver attaches the full command text as db.query.text to every database span by default, and Standard removes that attribute on the way out to the exporter. Detailed simply stops removing it. So Detailed is not "turning on" a capture that was previously off — it is switching off a redaction, and the data was already in the process either way.

Two consequences worth knowing:

• The command text is the SQL as submitted. Parameter values supplied as query parameters are not inlined — you will see $1, not the value. Any literal written directly into the SQL text (including in a customer-authored SQL data source) appears verbatim.
• The database span's name is the constant postgresql (connection spans are named CONNECT <database>). It never contains SQL, at either detail level, so switching back to Standard leaves nothing query-shaped behind in span names.

Verified against the Npgsql version CxReports ships, with a live PostgreSQL connection and no instrumentation configuration.

Third-party instrumentation, and its limits

CxReports enables three instrumentation libraries in addition to its own spans and metrics: ASP.NET Core (incoming HTTP requests), HttpClient (outgoing HTTP calls), and Npgsql (PostgreSQL queries, traces only).

These two sources of telemetry give different guarantees, and it matters which one you're relying on:

• CxReports' own spans and metrics carry only attributes CxReports explicitly sets, from a fixed, reviewed allowlist. There is no code path where they can pick up arbitrary application data.
• Third-party instrumentation attaches whatever the library attaches. The clearest example is Npgsql, which puts the full SQL command text on every database span by default; nothing CxReports does prevents it being created, only exported (see Detail levels). CxReports strips a named list of known-sensitive keys from every span on the way out — full query strings, request/response bodies, any http.request.header.*/http.response.header.* key, and (at Standard) SQL query text and exception messages. This is a denylist, not a sandbox: it is only as complete as the list of keys we know to strip. A future OpenTelemetry semantic-convention rename, or a library version that starts attaching sensitive data under a key not on that list, would pass through unfiltered. We keep the list current against the instrumentation versions CxReports ships, but if you enable an additional instrumentation library yourself (a custom OTel exporter processor, a different HTTP client library, etc.), you are responsible for reviewing what it emits before pointing it at a shared or long-retention backend.

The denylist filters span attributes, not span events. The stripping step reads each span's attributes only; it does not inspect events attached to a span (an "exception" event, for example, can itself carry attributes). This is inert today: the ASP.NET Core and HttpClient instrumentation libraries CxReports enables default their own exception recording to off, and CxReports deliberately does not override library defaults (see Detail levels). CxReports' own exception recording (CxReportsTelemetry.RecordException) is unaffected — it is written to withhold exception.message at Standard itself, rather than relying on this filter to catch it afterward. But if you enable exception recording on a third-party instrumentation library (stock ASP.NET Core/HttpClient, or one you add yourself), any message text that library puts on that event is not filtered by anything CxReports does — assess that separately before turning it on.

What CxReports puts on outbound requests. Enabling telemetry adds W3C trace context — the traceparent header — to outgoing HTTP calls and to NATS messages between instances, so a request that crosses services stays one trace. That header carries a trace id, a span id and a sampling flag, and nothing else. OpenTelemetry baggage is deliberately not propagated: CxReports registers only the W3C trace-context propagator, so an inbound baggage header is neither read nor forwarded. This matters because data sources call endpoints you configure, including third-party APIs — a baggage relay would forward arbitrary caller-supplied key/value data to them, and the attribute denylist above would not see it, because it filters span attributes rather than outbound headers.

If your security review requires a stronger guarantee than "reviewed denylist, best-effort against future library changes," restrict what reaches your OTLP collector at the collector/network layer, or keep telemetry off for the instances that handle the most sensitive workspaces.

Cardinality: what belongs on a dashboard

CxReports' own metrics have a bounded, tested attribute set: outcome, output.format, source.type, delivery.type — small, fixed vocabularies, enforced by an automated guard test so a future change can't silently add a high-cardinality attribute to a metric. workspace.id, report.id, template.name, and user.id are never on a metric — deliberately, since each distinct value becomes its own permanently stored time series. Those identifiers live on spans instead, where they're sampled and inspected trace-by-trace rather than aggregated forever.

The ASP.NET Core and HttpClient metrics enabled alongside CxReports' own are bounded a different way: by OpenTelemetry's semantic conventions (route template, method, status code), not by anything CxReports controls. That's a reasonable bound in practice, but it isn't the same guarantee, and it isn't ours to enforce — if you build dashboards against those metrics, apply the same cardinality discipline you would with any other OTel-instrumented service.

Worked configurations

Grafana + Tempo/Mimir (or any OTLP-native backend)

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=cxreports

Datadog

Point at the Datadog Agent's OTLP ingest (or Datadog's OTLP intake directly):

OTEL_EXPORTER_OTLP_ENDPOINT=http://datadog-agent:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=cxreports

Elastic (APM Server / Elastic Cloud OTLP endpoint)

OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-deployment>.apm.<region>.aws.elastic-cloud.com:443
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <apm-secret-token>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=cxreports

In every case, service.instance.id on exported telemetry is the same instance identifier already used for High Availability instance tracking, so traces and metrics can be correlated back to a specific instance in a multi-instance deployment.