# Observability for Abblix OIDC Server

An operator gets paged at night: token requests have started failing. The dashboard built on `oidc.request.duration` shows a rising line for the `token` endpoint with outcome `refused` and `oauth.error` set to `invalid_grant`, while every other endpoint stays flat. In the tracing backend the failing requests show up as spans named `token` under the host's HTTP spans, each with status Error and the same error code. With debug logging on for the server's telemetry, a log line next to each span names the endpoint and the code and carries the span's trace id, so the operator moves from the graph to a trace and from the trace to the host's own request logs, where the client is identified.

Starting with version 3.0, Abblix OIDC Server emits these traces, metrics, and logs through the standard .NET diagnostics APIs, so they reach any OpenTelemetry exporter the host already uses.

## Turn it on {#turn-it-on}

The server publishes its traces and its metrics under one name, `OidcTelemetry.SourceName` from `Abblix.Oidc.Server.Features.Telemetry`, which is `Abblix.Oidc.Server`. With the `OpenTelemetry.Extensions.Hosting` package (1.9.0 or later, where `WithLogging` arrived) and the `OpenTelemetry.Instrumentation.AspNetCore` package, a host subscribes like this:

```csharp
services.AddLogging(logging => logging.AddFilter(OidcTelemetry.LogCategory, LogLevel.Debug));
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddSource(OidcTelemetry.SourceName))
    .WithMetrics(metrics => metrics.AddMeter(OidcTelemetry.SourceName))
    .WithLogging();
```

Add the exporters you use (OTLP, Prometheus, or another) to each builder as usual. The metrics builder needs one: no meter is listened to until a reader or an exporter is attached. The ASP.NET Core instrumentation matters for the traces, because the server's spans are children of the host's HTTP spans, and without it those parents are neither exported nor sampled.

Nothing else is required on the server's side: each endpoint registers its instrumentation when `AddOidcServices` registers the endpoint, and the meter is created through the host's `IMeterFactory`, so each service container gets its own. A source nobody subscribes to starts no spans and reads nothing to tag them with, and an instrument nobody subscribes to records nothing.

## Traces {#traces}

Every request that reaches an endpoint's handler produces one span, a child of the HTTP span the host's ASP.NET Core instrumentation starts for that request. The span is of kind Internal, is named after the endpoint, and carries its name in `oidc.endpoint`:

| Endpoint | Span name |
|---|---|
| Authorization | `authorize` |
| [Pushed authorization](https://www.abblix.com/en/docs/glossary-overview#par) | `par` |
| Token | `token` |
| UserInfo | `userinfo` |
| End session | `end_session` |
| [Check session](https://www.abblix.com/en/docs/glossary-overview#check-session-iframe) | `check_session` |
| Revocation | `revocation` |
| [Introspection](https://www.abblix.com/en/docs/glossary-overview#introspection-endpoint) | `introspection` |
| [Back-channel authentication](https://www.abblix.com/en/docs/glossary-overview#ciba) | `backchannel_authentication` |
| [Device authorization](https://www.abblix.com/en/docs/glossary-overview#device-authorization-grant) | `device_authorization` |
| [Dynamic client registration](https://www.abblix.com/en/docs/glossary-overview#dynamic-client-registration) | `register`, `read_client`, `update_client`, `remove_client` |
| [Discovery](https://www.abblix.com/en/docs/discovery-and-well-known-endpoints) | `configuration` |

The [JWKS](https://www.abblix.com/en/docs/glossary-overview#jwks) endpoint, which publishes the server's signing keys, has no span of its own: it serves a ready document, and the host sees it in its own HTTP spans and measurements. A request the framework rejects before the handler runs, such as one whose form cannot be bound, is answered with `invalid_request` and likewise appears only in the host's HTTP spans and measurements.

A span ends with status Ok when the request succeeds. A request the server refuses ends with status Error and the error code it was answered with in `oauth.error`. A request that fails with an exception ends with status Error and the exception's type in `error.type`, as OpenTelemetry's general conventions name it. Every endpoint span carries `oidc.tenant`, the `token` span carries `oauth.grant_type`, and the `authorize` span carries `oauth.response_type`.

A refusal is marked Error even when the protocol expects it. A device or back-channel client polling the token endpoint gets `authorization_pending` or `slow_down` until the user acts, and a silent renewal with `prompt=none` gets `login_required` or `interaction_required`. Leave these codes out of error-rate alerts and of tail-sampling rules that keep every error, or those rules keep every poll.

### Stage spans {#stage-spans}

Under each endpoint span the server opens child spans for the stages of the work, named after the stage and carrying it in `oidc.stage`:

| Stage | What it covers |
|---|---|
| `validation` | Checking the request before anything acts on it |
| `client_authentication` | Telling which client sent the request from the credentials it presents |
| `grant` | Finding what a token request's grant authorizes |
| `consent` | Finding what the end user has consented to |
| `processing` | Acting on a validated request |
| `issuance` | Minting the tokens a validated token request is answered with |
| `signing` | Signing a token |
| `storage` | Reading, writing, or removing an entry of the server's storage |

A stage span starts only inside an endpoint span of the server, so work outside a request, such as storage a background service reads, records none. A stage that refuses the request ends with status Error and carries no error code; the endpoint span carries the code. A stage that fails with an exception ends with status Error and the exception's type in `error.type`.

## Metrics {#metrics}

The server records seven instruments into its meter:

| Instrument | Type | Attributes | What it measures |
|---|---|---|---|
| `oidc.request.duration` | histogram, seconds | `oidc.endpoint`, `oidc.outcome`, `oauth.error`, `oidc.tenant` | How long an endpoint's handler spends on a request. Its count is the number of requests. |
| `oidc.tokens.issued` | counter | `oidc.token_type`, `oauth.grant_type`, `oidc.tenant` | Tokens handed out, counted when they are minted. |
| `oidc.token.signing.duration` | histogram, seconds | `oidc.signing.alg` | How long signing one token takes. |
| `oidc.clients.registered` | counter | `oidc.outcome` | Requests to register a client (`register`); reading, updating, and removing a client are not counted here. |
| `oidc.rate_limit.refusals` | counter | `oidc.endpoint`, `oidc.rate_limit.budget` | Requests refused because a caller spent its budget. |
| `oidc.license.refusals` | counter | `oidc.license.reason` | Requests the license refused. |
| `oidc.storage.operation.duration` | histogram, seconds | `oidc.storage.operation` | How long one call to the server's entity storage takes, inside a request or outside it, such as a background service's. |

Through a Prometheus exporter the names take Prometheus form, for example `oidc_request_duration_seconds` and `oidc_tokens_issued_total`.

Tokens are counted wherever the server hands them out: at the token endpoint, in a [CIBA](https://www.abblix.com/en/docs/glossary-overview#ciba) push delivery, and through the front channel of the authorization endpoint. A token is counted when it is minted, so a token a push delivery fails to bring to the client is counted too. A registration that ends in an exception is counted with outcome `failed`.

`oidc.request.duration` covers the handler's own work on one request. At the authorization endpoint, a request that sends the user to a sign-in or consent page ends as a success as soon as the redirect is chosen, so the time the user spends on those pages is not in it. The signing duration covers the signing itself; when the key is held by an external custodian, such as a key vault, the measurement is the round trip to it. A token that is not signed is not recorded.

The histograms advise their bucket boundaries in seconds. `oidc.request.duration` uses 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, and 10, from 5 ms to 10 s. `oidc.token.signing.duration` and `oidc.storage.operation.duration` use 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, and 1, from 0.1 ms to 1 s, so a key that signs in process and one held by a remote custodian both land in buckets that tell them apart. A view overrides any of them where a backend needs different ones.

With many tenants, `oidc.tenant` multiplies the series of `oidc.request.duration` and `oidc.tokens.issued`. Where that is more than the backend should hold, drop the attribute from the metrics with a view and keep per-tenant drill-down on the spans, which carry it too.

## Attributes {#attributes}

No attribute carries a token, a code, a secret, or anything about the end user, and none names a client or a user. Most take their values from a closed set; `error.type` and `oidc.tenant` are the two that do not.

- `oidc.endpoint` is one of the span names in the table above.
- `oidc.outcome` is `success`, `refused` (the server answered with an error), or `failed` (an exception).
- `oauth.error` is the error code a refused request was answered with, when it is one of the library's error codes, and `other` otherwise. A request refused for a spent budget is answered with HTTP 429 and recorded here with `temporarily_unavailable`; `oidc.rate_limit.refusals` tells it apart from a server that is genuinely unavailable.
- `error.type` is the full name of the type of the exception a request failed with. It is on the spans only.
- `oauth.grant_type` is the grant type of a token request, when it is one the server supports, compared exactly; otherwise the attribute is left off. Tokens the authorization endpoint hands out through the [front channel](https://www.abblix.com/en/docs/glossary-overview#front-channel-communication), the [hybrid flow](https://www.abblix.com/en/docs/glossary-overview#hybrid-flow)'s included, count as `implicit`, since the protocol names no grant type of its own for them.
- `oauth.response_type` is the response type of an authorization request, its values ordered and space-separated, when each value is one the protocol defines; otherwise the attribute is left off. It is on the spans only.
- `oidc.token_type` is `access_token`, `id_token`, or `refresh_token`.
- `oidc.signing.alg` is the [JWS](https://www.abblix.com/en/docs/glossary-overview#jws) algorithm a token is signed with.
- `oidc.rate_limit.budget` names the budget a refused caller spent: `Abblix.Oidc.Server.AuthenticationFailures.RateLimit`, `Abblix.Oidc.Server.Introspection.CallerRateLimit`, `Abblix.Oidc.Server.Revocation.CallerRateLimit`, or `other` for a budget of the host's own.
- `oidc.license.reason` is `issuer_not_allowed`, `issuer_limit`, or `client_limit`.
- `oidc.stage` is one of the stage names in the table above. It is on the stage spans.
- `oidc.storage.operation` is `get`, `set`, `set_if_absent`, `replace`, `extend`, or `remove`.
- `oidc.tenant` is the id of the tenant serving the request.

## Adding your own attributes {#adding-your-own-attributes}

A host adds attributes of its own to each endpoint span by registering an `IEndpointSpanEnricher`. The server calls it once per request, after tagging the span and before handling the request, and only when the span is recorded:

```csharp
public sealed class ScopeCountEnricher : IEndpointSpanEnricher
{
    public void Enrich(Activity span, string endpoint, object? request)
    {
        if (request is TokenRequest tokenRequest)
            span.SetTag("app.token.scope_count", tokenRequest.Scope.Length);
    }
}

services.AddSingleton<IEndpointSpanEnricher, ScopeCountEnricher>();
```

The request reaches an enricher before it is validated, so it holds whatever the caller sent, credentials included: the access token of a UserInfo request, the registration access token of a client configuration request. Copy nothing secret into an attribute. The attributes an enricher adds are the host's, outside the table above, and what they say about a client or an end user is the host's decision; this is where a client id goes when a host wants one on its spans. Every registered enricher runs with the lifetime it is registered with, so one registered per request is created for every request, recorded or not. An enricher that throws fails the request.

## Logs {#logs}

A refused request is logged at Debug level with the endpoint and the same error code the span and the metric carry, under event id 10900 and the category `OidcTelemetry.LogCategory`, which is `Abblix.Oidc.Server.Telemetry`. The filter in the setup above lowers the minimum level for that category to Debug; filtering by the constant keeps it on the same name the records are written under. Setting all of `Abblix.Oidc.Server` to Debug instead turns on the whole library's debug output. With the OpenTelemetry logging bridge in place (`WithLogging()` above), the record carries the trace id and span id of the request's span, so a backend links the log line to its trace. Under sampling, a record can point to a trace that was not kept.

## The Shared Signals transmitter {#shared-signals-transmitter}

A host running the [Shared Signals](https://www.abblix.com/en/docs/shared-signals-framework) transmitter receives its signals under a name of its own, `SharedSignalsTelemetry.SourceName` from `Abblix.SharedSignals.Telemetry`, which is `Abblix.SharedSignals`. It is a separate source and meter from the server's, so a host subscribes to both:

```csharp
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddSource(OidcTelemetry.SourceName)
        .AddSource(SharedSignalsTelemetry.SourceName))
    .WithMetrics(metrics => metrics
        .AddMeter(OidcTelemetry.SourceName)
        .AddMeter(SharedSignalsTelemetry.SourceName));
```

The transmitter records two spans. `ssf.transmit` covers one security event minted for one stream and put into its outbox, the verification and stream-updated events included; an event dispatched while a request is handled sits under that request's span. `ssf.push` covers one security event token posted to a receiver, until its answer is acted on. Push delivery runs on a timer outside any request, so each `ssf.push` span is the root of its own trace. A stream whose receiver address the delivery policy refuses records nothing, since nothing is posted to it.

Its three instruments:

| Instrument | Type | Attributes | What it measures |
|---|---|---|---|
| `ssf.events.transmitted` | counter | `ssf.event_type`, `ssf.tenant` | Security events put into a stream's outbox. |
| `ssf.push.deliveries` | counter | `ssf.push.outcome`, `ssf.tenant` | Security event tokens posted to receivers. |
| `ssf.push.delivery.duration` | histogram, seconds | `ssf.push.outcome`, `ssf.tenant` | One push, from posting the token until its answer is acted on, with the same bucket boundaries as `oidc.request.duration`. |

`ssf.push.outcome` is `delivered` when the receiver accepted the token, `refused` when it answered 400, whether its verdict drops the event or keeps it queued, and `failed` for any other status, no answer, or an answer the outbox could not record; a failed event stays queued. `ssf.event_type` is the event type the host dispatched: dispatching to every matching stream sends only the types a stream receives, while dispatching to one stream sends whatever type it is given. `ssf.tenant` is the id of the tenant the transmitter answers for, the same value as `oidc.tenant`, and absent on a transmitter whose identity names no tenant. A span that ends with an exception carries its full type name in `error.type` and never its message, which names the stream and the receiver's address. No span or measurement names a stream, a receiver's address, or a subject.

## What to alert on {#what-to-alert-on}

Alert on the error code, never on the text of a message. The codes in `oauth.error` and the values of `oidc.outcome` are stable identifiers; the wording of a message is not part of the contract.

- The share of `oidc.request.duration` with outcome `failed`, per endpoint. The measurement does not carry the exception type, so look at `error.type` on the failed spans before acting: a client that drops the connection mid-request shows up here too.
- The rate of `refused` with a given `oauth.error` for an endpoint, against its usual level, leaving out the codes the protocol expects. A jump in `invalid_client` at the token endpoint often follows a client secret rotated on one side only; a jump in `invalid_grant` often means codes or refresh tokens arrive late or are reused. Which client it is, the server's telemetry does not say: follow a trace id into the host's own request logs.
- At the authorization endpoint, errors travel back to the client through the browser, so the server's signals are often the only place they are seen. Watch `access_denied`, `invalid_request_uri`, and `server_error` there against their usual level.
- `oidc.rate_limit.refusals` per budget, against its usual level. A steady trickle on the authentication-failures budget is normal for a server reachable from the internet; a step change means a misconfigured client or an attack.
- Any `oidc.license.refusals`.
- The upper percentiles of `oidc.token.signing.duration`, when keys live in an external custodian.
- `ssf.push.deliveries` with outcome `failed` or `refused` against its usual level, when the host transmits security events: a rise means a receiver is down or rejects the transmitter's credentials, and its events are waiting in the outbox.

Set latency targets per endpoint, since the endpoints do very different work, from signing tokens to calling into the host.
