Skip to content

Observability for Abblix OIDC Server

An operator gets paged at night: token requests have started failing. The dashboard built on oidc.request.duration shows a rising line for the token endpoint with outcome refused and oauth.error set to invalid_grant, while every other endpoint stays flat. In the tracing backend the failing requests show up as spans named token under the host's HTTP spans, each with status Error and the same error code. With debug logging on for the server's telemetry, a log line next to each span names the endpoint and the code and carries the span's trace id, so the operator moves from the graph to a trace and from the trace to the host's own request logs, where the client is identified.

Starting with version 3.0, Abblix OIDC Server emits these traces, metrics, and logs through the standard .NET diagnostics APIs, so they reach any OpenTelemetry exporter the host already uses.

Turn it on

The server publishes its traces and its metrics under one name, OidcTelemetry.SourceName from Abblix.Oidc.Server.Features.Telemetry, which is Abblix.Oidc.Server. With the OpenTelemetry.Extensions.Hosting package (1.9.0 or later, where WithLogging arrived) and the OpenTelemetry.Instrumentation.AspNetCore package, a host subscribes like this:

C#
services.AddLogging(logging => logging.AddFilter(OidcTelemetry.LogCategory, LogLevel.Debug));
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddSource(OidcTelemetry.SourceName))
    .WithMetrics(metrics => metrics.AddMeter(OidcTelemetry.SourceName))
    .WithLogging();

Add the exporters you use (OTLP, Prometheus, or another) to each builder as usual. The metrics builder needs one: no meter is listened to until a reader or an exporter is attached. The ASP.NET Core instrumentation matters for the traces, because the server's spans are children of the host's HTTP spans, and without it those parents are neither exported nor sampled.

Nothing else is required on the server's side: each endpoint registers its instrumentation when AddOidcServices registers the endpoint, and the meter is created through the host's IMeterFactory, so each service container gets its own. A source nobody subscribes to starts no spans and reads nothing to tag them with, and an instrument nobody subscribes to records nothing.

Traces

Every request that reaches an endpoint's handler produces one span, a child of the HTTP span the host's ASP.NET Core instrumentation starts for that request. The span is of kind Internal, is named after the endpoint, and carries its name in oidc.endpoint:

EndpointSpan name
Authorizationauthorize
Pushed authorizationpar
Tokentoken
UserInfouserinfo
End sessionend_session
Check sessioncheck_session
Revocationrevocation
Introspectionintrospection
Back-channel authenticationbackchannel_authentication
Device authorizationdevice_authorization
Dynamic client registrationregister, read_client, update_client, remove_client
Discoveryconfiguration

The JWKS endpoint, which publishes the server's signing keys, has no span of its own: it serves a ready document, and the host sees it in its own HTTP spans and measurements. A request the framework rejects before the handler runs, such as one whose form cannot be bound, is answered with invalid_request and likewise appears only in the host's HTTP spans and measurements.

A span ends with status Ok when the request succeeds. A request the server refuses ends with status Error and the error code it was answered with in oauth.error. A request that fails with an exception ends with status Error and the exception's type in error.type, as OpenTelemetry's general conventions name it. Every endpoint span carries oidc.tenant, the token span carries oauth.grant_type, and the authorize span carries oauth.response_type.

A refusal is marked Error even when the protocol expects it. A device or back-channel client polling the token endpoint gets authorization_pending or slow_down until the user acts, and a silent renewal with prompt=none gets login_required or interaction_required. Leave these codes out of error-rate alerts and of tail-sampling rules that keep every error, or those rules keep every poll.

Stage spans

Under each endpoint span the server opens child spans for the stages of the work, named after the stage and carrying it in oidc.stage:

StageWhat it covers
validationChecking the request before anything acts on it
client_authenticationTelling which client sent the request from the credentials it presents
grantFinding what a token request's grant authorizes
consentFinding what the end user has consented to
processingActing on a validated request
issuanceMinting the tokens a validated token request is answered with
signingSigning a token
storageReading, writing, or removing an entry of the server's storage

A stage span starts only inside an endpoint span of the server, so work outside a request, such as storage a background service reads, records none. A stage that refuses the request ends with status Error and carries no error code; the endpoint span carries the code. A stage that fails with an exception ends with status Error and the exception's type in error.type.

Metrics

The server records seven instruments into its meter:

InstrumentTypeAttributesWhat it measures
oidc.request.durationhistogram, secondsoidc.endpoint, oidc.outcome, oauth.error, oidc.tenantHow long an endpoint's handler spends on a request. Its count is the number of requests.
oidc.tokens.issuedcounteroidc.token_type, oauth.grant_type, oidc.tenantTokens handed out, counted when they are minted.
oidc.token.signing.durationhistogram, secondsoidc.signing.algHow long signing one token takes.
oidc.clients.registeredcounteroidc.outcomeRequests to register a client (register); reading, updating, and removing a client are not counted here.
oidc.rate_limit.refusalscounteroidc.endpoint, oidc.rate_limit.budgetRequests refused because a caller spent its budget.
oidc.license.refusalscounteroidc.license.reasonRequests the license refused.
oidc.storage.operation.durationhistogram, secondsoidc.storage.operationHow long one call to the server's entity storage takes, inside a request or outside it, such as a background service's.

Through a Prometheus exporter the names take Prometheus form, for example oidc_request_duration_seconds and oidc_tokens_issued_total.

Tokens are counted wherever the server hands them out: at the token endpoint, in a CIBA push delivery, and through the front channel of the authorization endpoint. A token is counted when it is minted, so a token a push delivery fails to bring to the client is counted too. A registration that ends in an exception is counted with outcome failed.

oidc.request.duration covers the handler's own work on one request. At the authorization endpoint, a request that sends the user to a sign-in or consent page ends as a success as soon as the redirect is chosen, so the time the user spends on those pages is not in it. The signing duration covers the signing itself; when the key is held by an external custodian, such as a key vault, the measurement is the round trip to it. A token that is not signed is not recorded.

The histograms advise their bucket boundaries in seconds. oidc.request.duration uses 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, and 10, from 5 ms to 10 s. oidc.token.signing.duration and oidc.storage.operation.duration use 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, and 1, from 0.1 ms to 1 s, so a key that signs in process and one held by a remote custodian both land in buckets that tell them apart. A view overrides any of them where a backend needs different ones.

With many tenants, oidc.tenant multiplies the series of oidc.request.duration and oidc.tokens.issued. Where that is more than the backend should hold, drop the attribute from the metrics with a view and keep per-tenant drill-down on the spans, which carry it too.

Attributes

No attribute carries a token, a code, a secret, or anything about the end user, and none names a client or a user. Most take their values from a closed set; error.type and oidc.tenant are the two that do not.

  • oidc.endpoint is one of the span names in the table above.
  • oidc.outcome is success, refused (the server answered with an error), or failed (an exception).
  • oauth.error is the error code a refused request was answered with, when it is one of the library's error codes, and other otherwise. A request refused for a spent budget is answered with HTTP 429 and recorded here with temporarily_unavailable; oidc.rate_limit.refusals tells it apart from a server that is genuinely unavailable.
  • error.type is the full name of the type of the exception a request failed with. It is on the spans only.
  • oauth.grant_type is the grant type of a token request, when it is one the server supports, compared exactly; otherwise the attribute is left off. Tokens the authorization endpoint hands out through the front channel, the hybrid flow's included, count as implicit, since the protocol names no grant type of its own for them.
  • oauth.response_type is the response type of an authorization request, its values ordered and space-separated, when each value is one the protocol defines; otherwise the attribute is left off. It is on the spans only.
  • oidc.token_type is access_token, id_token, or refresh_token.
  • oidc.signing.alg is the JWS algorithm a token is signed with.
  • oidc.rate_limit.budget names the budget a refused caller spent: Abblix.Oidc.Server.AuthenticationFailures.RateLimit, Abblix.Oidc.Server.Introspection.CallerRateLimit, Abblix.Oidc.Server.Revocation.CallerRateLimit, or other for a budget of the host's own.
  • oidc.license.reason is issuer_not_allowed, issuer_limit, or client_limit.
  • oidc.stage is one of the stage names in the table above. It is on the stage spans.
  • oidc.storage.operation is get, set, set_if_absent, replace, extend, or remove.
  • oidc.tenant is the id of the tenant serving the request.

Adding your own attributes

A host adds attributes of its own to each endpoint span by registering an IEndpointSpanEnricher. The server calls it once per request, after tagging the span and before handling the request, and only when the span is recorded:

C#
public sealed class ScopeCountEnricher : IEndpointSpanEnricher
{
    public void Enrich(Activity span, string endpoint, object? request)
    {
        if (request is TokenRequest tokenRequest)
            span.SetTag("app.token.scope_count", tokenRequest.Scope.Length);
    }
}

services.AddSingleton<IEndpointSpanEnricher, ScopeCountEnricher>();

The request reaches an enricher before it is validated, so it holds whatever the caller sent, credentials included: the access token of a UserInfo request, the registration access token of a client configuration request. Copy nothing secret into an attribute. The attributes an enricher adds are the host's, outside the table above, and what they say about a client or an end user is the host's decision; this is where a client id goes when a host wants one on its spans. Every registered enricher runs with the lifetime it is registered with, so one registered per request is created for every request, recorded or not. An enricher that throws fails the request.

Logs

A refused request is logged at Debug level with the endpoint and the same error code the span and the metric carry, under event id 10900 and the category OidcTelemetry.LogCategory, which is Abblix.Oidc.Server.Telemetry. The filter in the setup above lowers the minimum level for that category to Debug; filtering by the constant keeps it on the same name the records are written under. Setting all of Abblix.Oidc.Server to Debug instead turns on the whole library's debug output. With the OpenTelemetry logging bridge in place (WithLogging() above), the record carries the trace id and span id of the request's span, so a backend links the log line to its trace. Under sampling, a record can point to a trace that was not kept.

The Shared Signals transmitter

A host running the Shared Signals transmitter receives its signals under a name of its own, SharedSignalsTelemetry.SourceName from Abblix.SharedSignals.Telemetry, which is Abblix.SharedSignals. It is a separate source and meter from the server's, so a host subscribes to both:

C#
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddSource(OidcTelemetry.SourceName)
        .AddSource(SharedSignalsTelemetry.SourceName))
    .WithMetrics(metrics => metrics
        .AddMeter(OidcTelemetry.SourceName)
        .AddMeter(SharedSignalsTelemetry.SourceName));

The transmitter records two spans. ssf.transmit covers one security event minted for one stream and put into its outbox, the verification and stream-updated events included; an event dispatched while a request is handled sits under that request's span. ssf.push covers one security event token posted to a receiver, until its answer is acted on. Push delivery runs on a timer outside any request, so each ssf.push span is the root of its own trace. A stream whose receiver address the delivery policy refuses records nothing, since nothing is posted to it.

Its three instruments:

InstrumentTypeAttributesWhat it measures
ssf.events.transmittedcounterssf.event_type, ssf.tenantSecurity events put into a stream's outbox.
ssf.push.deliveriescounterssf.push.outcome, ssf.tenantSecurity event tokens posted to receivers.
ssf.push.delivery.durationhistogram, secondsssf.push.outcome, ssf.tenantOne push, from posting the token until its answer is acted on, with the same bucket boundaries as oidc.request.duration.

ssf.push.outcome is delivered when the receiver accepted the token, refused when it answered 400, whether its verdict drops the event or keeps it queued, and failed for any other status, no answer, or an answer the outbox could not record; a failed event stays queued. ssf.event_type is the event type the host dispatched: dispatching to every matching stream sends only the types a stream receives, while dispatching to one stream sends whatever type it is given. ssf.tenant is the id of the tenant the transmitter answers for, the same value as oidc.tenant, and absent on a transmitter whose identity names no tenant. A span that ends with an exception carries its full type name in error.type and never its message, which names the stream and the receiver's address. No span or measurement names a stream, a receiver's address, or a subject.

What to alert on

Alert on the error code, never on the text of a message. The codes in oauth.error and the values of oidc.outcome are stable identifiers; the wording of a message is not part of the contract.

  • The share of oidc.request.duration with outcome failed, per endpoint. The measurement does not carry the exception type, so look at error.type on the failed spans before acting: a client that drops the connection mid-request shows up here too.
  • The rate of refused with a given oauth.error for an endpoint, against its usual level, leaving out the codes the protocol expects. A jump in invalid_client at the token endpoint often follows a client secret rotated on one side only; a jump in invalid_grant often means codes or refresh tokens arrive late or are reused. Which client it is, the server's telemetry does not say: follow a trace id into the host's own request logs.
  • At the authorization endpoint, errors travel back to the client through the browser, so the server's signals are often the only place they are seen. Watch access_denied, invalid_request_uri, and server_error there against their usual level.
  • oidc.rate_limit.refusals per budget, against its usual level. A steady trickle on the authentication-failures budget is normal for a server reachable from the internet; a step change means a misconfigured client or an attack.
  • Any oidc.license.refusals.
  • The upper percentiles of oidc.token.signing.duration, when keys live in an external custodian.
  • ssf.push.deliveries with outcome failed or refused against its usual level, when the host transmits security events: a rise means a receiver is down or rejects the transmitter's credentials, and its events are waiting in the outbox.

Set latency targets per endpoint, since the endpoints do very different work, from signing tokens to calling into the host.