Persisting JWT Signing Keys in Production

Overview

Abblix OIDC Server signs JWT tokens using keys supplied through the IAuthServiceKeysProvider contract. Out of the box it reads them from OidcOptions.SigningKeys, and with that collection empty the server refuses to start. The trap is the quick workaround: generating a key at startup. In any multi-instance deployment or across process restarts, such keys produce tokens that other instances or future restarts cannot validate. The result is intermittent invalid_token errors and mass logouts under rolling deploys.

This guide covers the IAuthServiceKeysProvider contract, the three production failure modes caused by ephemeral or per-instance keys, four storage recipes in order of preference (an HSM or KMS, a static secret, Redis, a database), encrypting the stored key at rest, the key rotation discipline, and a verification test suitable for CI.

Scope: signing keys, not Data Protection

Before anything else, a disambiguation worth a minute of your time.

ASP.NET Core Data Protection is the framework service that protects cookies, anti-forgery tokens, and TempData. Its key storage is a separate concern, configured via services.AddDataProtection().PersistKeysTo...(). If you've read that .NET teams routinely lose production Data Protection keys on restart and wonder whether that applies to Abblix, the answer is: it does not, because the library does not route signing through Data Protection.

Abblix OIDC Server signing keys are the JSON Web Keys (JWKs) used to sign ID tokens, access tokens, and refresh tokens. They live behind a dedicated library contract, IAuthServiceKeysProvider.

Both problems have the same shape (a key store that must survive restarts and be shared across pods) and both failure modes look identical from the outside (random authentication errors, phantom logouts). But the remedies are unrelated. Configuring Data Protection persistence does not persist your signing keys. You have to do both.

The rest of this article is about the signing-key side. Data Protection comes back later only as a way to encrypt that key at rest, not in its usual role of protecting cookies and anti-forgery tokens.

The library contract

Abblix OIDC Server exposes signing-key access through a small interface. IAuthServiceKeysProvider has two methods, GetSigningKeys and GetEncryptionKeys, each returning an IAsyncEnumerable<JsonWebKey> and each taking one flag, includePrivateKeys, which defaults to false: the public half is what the JWKS endpoint publishes, the private half is what the signer asks for.

public interface IAuthServiceKeysProvider
{
    IAsyncEnumerable<JsonWebKey> GetEncryptionKeys(bool includePrivateKeys = false);
    IAsyncEnumerable<JsonWebKey> GetSigningKeys(bool includePrivateKeys = false);
}

A key custodian changes the second half: with keys held in a vault there is no private half to hand out, and each signature is made by the vault (see Keys in an HSM or KMS).

IAsyncEnumerable is deliberate. Retrieval may hit a cache, a Redis hash, a database table, or an external KMS. The library does not assume synchronous in-memory access.

The default implementation is OidcOptionsKeysProvider, which reads from IOptions<OidcOptions> - two collections of JsonWebKey, SigningKeys and EncryptionKeys.

The default value of both properties is an empty collection, and the server will not start without a signing key. The library offers three routes, a key custodian for a vault, the options for static keys and the interface for a store of your own, and expects the host to choose one before going live.

The XML documentation on OidcOptionsKeysProvider says this out loud:

It is recommended to implement a dynamic resolution mechanism in production environments to enable seamless certificate replacement without the need for service reloading.

Production failure modes

Three concrete scenarios motivate the recipes in the rest of this guide. Each produces authentication errors that look random until you know the cause.

Multi-pod key divergence

A Kubernetes Deployment runs three pods of your auth-service. Each pod's startup code calls something like JsonWebKeyFactory.CreateRsa() to populate OidcOptions.SigningKeys. Each pod independently generates a different key pair. The three pods now have three different jwks_uri responses, three different sets of kid values, and three different private keys signing tokens that are all presented to the same load balancer.

The user impact is intermittent 401 errors that correlate with nothing the user can control. Sticky sessions partially mask the problem; a rolling deploy exposes it. Your tracing shows nothing in application code. The only clue is that failures correlate with the kid claim on the token not matching the verifying pod's published JWKS.

Diagnosis is slow because the usual suspects (token expiry, clock skew, incorrect audience) all check out.

Restart invalidates every token

A single-pod deployment has keys generated at process start. Every refresh token in circulation is signed by the current process's in-memory private key. A rolling restart, a liveness-probe kill, or a Kubernetes eviction replaces the process. New keys, new kid values, and every refresh token that was valid a minute ago is now unrecognized.

Users experience a mass logout. Mobile apps start retrying refresh, hit the auth endpoint harder than steady state, and some teams accidentally DoS themselves during a routine deploy.

Keys in Git

The third failure mode is avoided by engineers who know not to commit .pem files, but it keeps appearing in leaked-secrets audits anyway. A developer pastes a JWK into appsettings.Development.json, copies that file into appsettings.Production.json to "unblock staging", and a month later somebody greps the repo history and finds a live production private key in a commit.

The code works. The deploy works. The security review three quarters later finds the key.

Choosing a recipe

The four recipes below go from most to least preferred: keys in an HSM or KMS, static keys from a shared secret, a Redis-backed provider, a database-backed provider. Take the first one your environment allows (keys in an HSM or KMS need 2.4 or later).

The order follows where the private key rests and who rotates it. A vault either never lets the key out or holds the key that unseals it, and rotation is built in. A secret store hands the key to the process at startup, and you rotate by hand. Redis and a database keep it next to the rest of your application's data, with rotation code you write yourself.

Keys in an HSM or KMS

The signing key lives in an HSM or a managed KMS, and the library reaches it through a key custodian. Every pod uses the same vault and a restart loses nothing, which removes the first two failure modes. The third one moves rather than disappears: there is no key file to commit, but the credential that reaches the vault can have it sign for as long as that credential is valid. Keep it out of source control as you would a key.

Two custodian packages ship with 2.4. Abblix.JWT.Vault works with HashiCorp Vault and OpenBao Transit, Abblix.JWT.Azure with Azure Key Vault, including HSM-backed keys. For AWS KMS, GCP KMS or any other vault, register your own implementation of IKeyCustodian with AddCustodian<T>.

Wiring

Register the custodian after AddOidcServices(), then say where the private halves live. That second call is required: a custodian registered without it keeps the server from starting, so a half-wired setup cannot quietly fall back to local keys.

dotnet add package Abblix.JWT.Vault

using Abblix.Jwt.ExternalKeys;
using Abblix.Jwt.Vault;

services.AddOidcServices(options => { /* ... */ });

services
    .AddVaultCustodian(vault => configuration.GetSection("Vault").Bind(vault))
    .UseKeysInCustodian(new CustodianHeldKeys { SigningKeyName = "oidc-sign" });

"Vault": {
  "Address": "https://vault.internal:8200",
  "Authentication": { "Kubernetes": { "Role": "oidc-signer" } }
}

With the Authentication section the package logs in by itself and renews its own short-lived token, so no long-lived token has to reach the process. AppRole works the same way outside Kubernetes. For Azure the shape is the same, with AddAzureCustodian, the vault URI and, in production, a managed identity. Reach the vault over TLS. The package READMEs cover the remaining authentication options.

Two placements

UseKeysInCustodian and UseKeysInProcess decide whether the private half ever enters your process.

• Keys held in the vault, UseKeysInCustodian. The vault creates the key and never exports it, and every signature is made there. A compromised server process has no key to leak, though it can still ask the vault to sign until its credential expires, so scope that credential narrowly and watch the vault's audit log: it is the record of what was signed.

  The price is the vault on your hot path. Each issued token costs at least two vault round-trips (listing the key's versions, then signing), and on Azure one more per enabled key version. Endpoints that check tokens this server issued, such as userinfo, introspection and the refresh grant, list the versions too. A vault that is unreachable, sealed or throttling stops all of them, so size it for your peak token rate and set an explicit HTTP timeout.

• Keys minted in process, UseKeysInProcess. The server creates its own keys, seals each one to a key-encryption key in the vault, keeps the sealed copies in a store and signs locally, so a token costs no vault call. This placement takes a third call that names the store, PersistRingToVaultKeyValue or PersistRingToAzureBlob. Rotation runs on the RotateEvery schedule with no manual step. The tradeoff: a compromised process can reach every key the pod has unsealed, and the vault sees only unwraps, never the signatures. A pod that starts while the vault is down cannot unseal its keys, and neither can a pod that did not mint the newest one, so plan outage tolerance around the rotation interval.

The keys provider that serves a custodian does not cache, and the JWKS endpoint is public, so without a cache every scraper hit reaches the vault. Register your own IAuthServiceKeysProvider that wraps the library's provider (ActivatorUtilities.CreateInstance<ExternalKeysProvider>(sp)) and caches its key list for a few seconds, well under the rollover window described below. Cache only successful reads, so a short vault outage is not pinned for the whole lifetime of the entry. The library adds its own provider only when none is registered, so yours takes its place.

Provisioning

For keys held in Vault Transit, enable the engine once and create the signing key. It stays non-exportable by default:

vault secrets enable transit
vault write -f transit/keys/oidc-sign type=rsa-2048

On Azure the equivalent is az keyvault key create, with --kty RSA-HSM on a Premium vault for an HSM-backed key. With keys held in the vault, the server's credential needs only to read the key and sign, plus decrypt if you configure an encryption key. With minted keys it needs read and unwrap on the key-encryption key, plus read, write, list and delete on the store. Neither placement ever creates, exports or deletes a vault key. The package READMEs list the exact policy and role for each placement.

Rotation

Rotation needs no redeploy in either placement. With keys held in the vault you rotate there (vault write -f transit/keys/oidc-sign/rotate). Every version is published under its own kid, and a new version starts signing only after OidcOptions.KeyRolloverPropagation has passed. Set that window to at least your slowest client's JWKS cache lifetime, and a client never encounters a kid it has not fetched. You still retire old versions yourself: delete or disable a version only after every token it signed has expired, refresh tokens included.

With minted keys the server mints, publishes, switches and retires on its own. Its rollover window is KeyRingOptions.KeyRolloverPropagation, and in 2.4 it is not taken from OidcOptions, so set both to the same value (services.Configure<KeyRingOptions>(...)). Retired keys stay published for KeepRetiredFor, which must outlast your longest refresh token.

Moving an existing deployment

Switching to a custodian changes the published key set, and done naively it logs everyone out: the current key leaves JWKS, so every refresh token it signed stops validating, and a vault key with a single version signs from its first second. With minted keys, chain AdoptExistingKeys(currentKey) before the PersistRingTo... call. The current key keeps signing while the first minted key serves out its rollover window, then stays published until it retires. With keys held in the vault, keep publishing the current key yourself, through your own provider, until every token it signed has expired.

EXTERNAL_KEYS.md explains both placements in full: what each protects against, what it costs, and how rotation works.

Runnable sample

The ExternalKeySample project of the Getting Started repository runs both placements against OpenBao in Docker or against Azure Key Vault. It issues and introspects tokens, and sealing OpenBao shows what a vault outage does to each placement: with keys held in the vault, signing stops at once; with minted keys, a running server keeps signing and a fresh one cannot start.

Static keys from a shared secret

When no HSM or KMS can hold the key, this is the simplest production-grade setup. It fits when you have a small number of pods, a stable key set, and no strict rotation SLA.

Keys live in a secret manager (a Kubernetes Secret, a key-value engine such as Vault KV, or AWS Secrets Manager) and are mounted into every pod as an environment variable or file. All pods read the same keys at startup and never change them until a planned rotation.

Generate a JWK once, store it in your secret manager, and load it through configuration: inside the AddOidcServices options delegate, read the environment variable holding the JWK - ABBLIX_SIGNING_JWK below - deserialize it into a JsonWebKey, and assign it to options.SigningKeys. Read both steps as mandatory - a missing variable and unparsable JSON each throw at startup with a message that names the variable.

services.AddOidcServices(options =>
{
    var jwkJson = Environment.GetEnvironmentVariable("ABBLIX_SIGNING_JWK")
        ?? throw new InvalidOperationException("ABBLIX_SIGNING_JWK is required.");

    var jwk = JsonSerializer.Deserialize<JsonWebKey>(jwkJson)
        ?? throw new InvalidOperationException("ABBLIX_SIGNING_JWK is not valid JSON.");

    options.SigningKeys = new[] { jwk };
});

The corresponding secret (Kubernetes example):

kubectl create secret generic abblix-signing-key \
  --from-literal=ABBLIX_SIGNING_JWK="$(cat signing-key.jwk.json)" \
  --namespace=production

And the Deployment references it:

env:
  - name: ABBLIX_SIGNING_JWK
    valueFrom:
      secretKeyRef:
        name: abblix-signing-key
        key: ABBLIX_SIGNING_JWK

What this buys you:

• Every pod sees the same keys at startup. No multi-pod divergence.
• A pod restart reads the same secret and produces the same jwks_uri. Existing tokens remain valid.
• The key never appears in git or in a built image. It does live in process memory, as it does in every recipe except keys held in a vault.

What this does not buy you:

• Automatic rotation. A new key requires a manual secret update and a restart.
• Graceful overlap during rotation. Unless you deliberately publish both the old and the new key for a window, clients will see JWKS churn and existing tokens may stop validating.

For teams with low-volume auth traffic and rotations measured in months, this recipe is often the right answer. A compliance-driven rotation policy is better served by keys in an HSM or KMS; without a vault, the Redis and database recipes below rotate without a deploy.

Redis-backed provider

When rotation is frequent, or when you want the ability to swap keys without a deploy, a Redis-backed provider gives you a shared, out-of-process key store that every pod reads from on demand.

The library deliberately does not depend on any particular Redis client, so the host owns the dependency. That keeps Abblix OIDC Server free of transitive weight for teams who use a different backend, and it keeps version conflicts from forcing your hand.

The shape of such a provider, given a host that has StackExchange.Redis available and an IConnectionMultiplexer registered: a class implementing IAuthServiceKeysProvider whose two methods differ only in which hash they read, both delegating to one private reader. That reader fetches the whole hash, deserializes each field value into a JsonWebKey, logs and skips a malformed entry instead of failing the whole read, and yields each key through Sanitize(includePrivateKeys) - the call that strips the private half unless the caller asked for it, so the same store serves the JWKS endpoint and the signer.

Register it as shown in Registering your provider, with SigningKeys in the options left empty.

What to put in Redis:

• A hash per key-usage (one for signing, one for encryption).
• Each field is a kid, each value is the full JWK serialized as JSON.
• A separate small wrapper job (or an admin UI) adds new keys and retires old ones on schedule.

Performance note: a production-quality Redis provider caches the result of that read in-process for a short TTL (five to thirty seconds is typical). Reading Redis for every token-sign operation is unnecessary overhead, because keys rotate on human timescales, not per-request. Implement the cache with IMemoryCache or equivalent, and invalidate on a simple Redis pub/sub signal if you need sub-minute propagation.

The runnable sample in the repository is the database-backed recipe below, not this one; the Redis shape mirrors it, swapping the table query for a hash read.

Database-backed provider

The library does not prescribe an ORM, and neither does this recipe. If you already have EF Core, use EF Core. If you prefer Dapper, use Dapper. If you're writing raw ADO.NET against a legacy stored-procedure layer, that works too. What matters is the table shape and the query.

Suggested table:

CREATE TABLE abblix_signing_keys (
    kid          TEXT PRIMARY KEY,
    key_usage    TEXT NOT NULL,          -- 'sig' or 'enc'
    algorithm    TEXT NOT NULL,          -- e.g. 'RS256', 'ES384'
    jwk_json     TEXT NOT NULL,          -- full JWK, including private material
    not_before   TIMESTAMPTZ,            -- NULL means effective immediately
    not_after    TIMESTAMPTZ,            -- NULL means indefinite
    is_active    BOOLEAN NOT NULL DEFAULT TRUE,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX abblix_signing_keys_active
    ON abblix_signing_keys (key_usage, not_after)
    WHERE is_active = TRUE;

The effective query for GetSigningKeys:

SELECT jwk_json
  FROM abblix_signing_keys
 WHERE key_usage = 'sig'
   AND is_active = TRUE
   AND (not_after IS NULL OR not_after > NOW())
 ORDER BY CASE WHEN not_before IS NULL OR not_before <= NOW() THEN 0 ELSE 1 END,
          created_at DESC;

The ordering matters. The server publishes every key the provider returns and signs with the first one for each algorithm. A key whose not_before has not arrived is therefore returned, so it is published early, but after the active key, so it does not sign yet.

Wrap that in whichever data-access technology you already use. Cache the result in-process with the same short TTL you'd use for the Redis recipe. Implement IAuthServiceKeysProvider on top of the query-plus-cache and register it as shown in Registering your provider.

Private keys at rest deserve their own conversation, and the honest answer is that an application database is not their real home. In a high-assurance system signing keys live in an HSM or a managed KMS (HashiCorp Vault or OpenBao and Azure Key Vault through the shipped packages, AWS KMS, GCP KMS or anything else through your own custodian): see Keys in an HSM or KMS. The database-backed recipe here is a documented, deliberate tradeoff for teams that accept DB key custody. If you take it, encrypt the key at rest at a minimum, either at the database layer (pgcrypto, Always Encrypted, TDE with additional protections) or at the application layer with ASP.NET Data Protection (see Encrypting the stored key at rest). Whichever layer you pick, the encryption relocates key custody to that layer's own master key rather than removing it, as the next section traces for Data Protection. Left unencrypted and unrotated, a database holding raw RSA private keys is worse than a Kubernetes Secret.

A runnable version of this recipe, backed by EF Core and SQLite, lives in the AspNetIdentitySample project of the Getting Started repository. It generates and persists a signing key on first run, encrypts it at rest with ASP.NET Data Protection (see the next section), and reloads the same key on every restart, so tokens issued before a restart keep validating. The residual custody concerns are marked with #warning directives in the sample, so a developer copying it toward production trips over them at build time.

Encrypting the stored key at rest

Storing a private key as plaintext in any database is the weakest form of this recipe. One step up, without reaching for a KMS, is to encrypt the JWK before it is written and decrypt it on read. ASP.NET Core Data Protection is the built-in tool for that: create a protector with a fixed purpose string, Protect the serialized JWK on the way in, Unprotect it on the way out. The protector comes from IDataProtectionProvider.CreateProtector, and the purpose string you pass it is part of the key derivation: change it and everything already stored stops decrypting, so pick one and version it deliberately.

The AspNetIdentitySample sample does this, so a raw dump of the SQLite file yields ciphertext, not a signing key. Be precise, though, about what this buys and what it does not.

There is an irony worth naming. The scope note at the top of this article warned that Data Protection keys and signing keys are two separate stores with the same failure modes. Encrypting your signing key with Data Protection does not merge them, it stacks them: your signing key is now exactly as durable and as shared as the Data Protection key ring that encrypts it. The custody problem moves one layer down, onto a key ring that must be persisted and shared exactly as this article demands of the signing key, and protected on top of that: a smaller, better-understood surface to defend, but not one you can ignore.

• Persisted. With no explicit configuration the key ring lands in a per-user profile folder on a developer machine, which is exactly why a local restart still decrypts the signing key. Inside a container with no user profile that fallback is not available, and Data Protection degrades to ephemeral in-memory keys lost on the next restart. Persist and back up the ring, or a lost ring leaves every stored signing key unrecoverable. Call PersistKeysToFileSystem(...) with a durable location (or a shared store) and SetApplicationName(...) so every instance derives the same keys, rather than relying on the developer-machine default.
• Shared. Two instances with two different key rings cannot read each other's encrypted signing keys, so an unshared ring has the same multi-instance problem this article opened with, one layer down. Persist the ring to a store every instance reads, using the backend-specific extension: PersistKeysToFileSystem for a shared volume (in-box), PersistKeysToStackExchangeRedis, PersistKeysToAzureBlobStorage, or PersistKeysToDbContext, each in its own package. There is no generic PersistKeysTo.
• Protected. A persisted ring is itself plaintext at rest unless you protect it with ProtectKeysWithCertificate(...) or a cloud KMS (ProtectKeysWithAzureKeyVault(...)). Without that, the Data Protection keys sit next to the database you were protecting, and the encryption defends only against a leaked database file, not against read access to the host filesystem. The certificate option only defers custody again (the cert's own private key must be loadable to open the ring); a cloud KMS is what actually terminates the chain.

If you keep the key in a database, by deployment shape:

• Single instance, development or low assurance. Data Protection with PersistKeysToFileSystem pointed at a durable path is enough. Know that without ProtectKeysWith, you are defended against a leaked database backup but not against an attacker who can read the host filesystem.
• Multiple instances. The key ring must be both shared (a store every instance reads: a mounted volume, a blob container, Redis, a database table) and protected (a certificate or KMS). Configure both: miss the shared store and validation fails across pods, miss the protector and the encryption adds nothing over the database's own protection.
• Compliance-bound, or able to run a vault. Leave the database and use keys in an HSM or KMS. Data Protection encryption is a pragmatic middle ground, not a substitute.

Key rotation discipline

With keys in an HSM or KMS most of this is built in: minted keys go through all four moves on their own, and with keys held in the vault you trigger the rotation and retire old versions while the server handles the delay between publishing and signing (see Rotation). A key in your own store needs all four moves from you. They are written below in terms of the database recipe's columns. With Redis, keep the activation and retirement times next to each key, for example in a second hash. With a static secret, store both keys in the secret as a JSON array for the overlap window, deserialize it into JsonWebKey[], list the key that should sign first, and restart.

• Generate a new key ahead of time. Give it a unique kid, a not_before set to about ten minutes from now, and insert it as is_active = TRUE. Your JWKS endpoint will start publishing the new key immediately, but the signing pipeline will keep using the previous one until the not_before window opens.
• Start signing with the new key. At the configured not_before, your provider returns the new key first, and the server signs with it. Relying parties who fetched JWKS in the last ten minutes already have the new kid in their cache. Freshly issued tokens validate.
• Retain the old key for the token lifetime plus a buffer. Set the old key's not_after to now plus your longest refresh-token lifetime plus a safety margin of at least an hour. During this overlap window, both keys are published in JWKS, old tokens validate against the old key, and new tokens validate against the new key. Nobody is locked out.
• Retire the old key. Once no extant token could still be signed by it, set is_active = FALSE and drop it from JWKS. Keep the row around for an audit trail, but stop publishing the public half.

A common beginner mistake is to rotate the key and immediately remove the old one, which invalidates every outstanding refresh token. Another is to skip not_before and start signing with a key the JWKS endpoint hasn't published yet, which causes a brief window of 401s for every client that cached JWKS too recently. The overlap window is not a nicety, it is the whole point.

A multi-pod consistency test worth writing

An integration test you can run in CI, with throwaway keys in a shared key source:

• Start two auth-service instances against the same source: the same vault (a dev-mode Vault or OpenBao container is enough), the same secret, or the same Redis or database.
• Issue a token from instance A.
• Present that token to instance B's token introspection endpoint.
• Assert the introspection result is active: true.

Repeat with a forced key rotation between issue and validate. Assert the old-signed token still validates during the overlap window. Assert it fails after the window elapses.

This test catches almost every real configuration mistake in this area before it reaches production. It runs in CI alongside normal integration tests.

Registering your provider

The Redis and database recipes bring their own IAuthServiceKeysProvider, and so does the caching wrapper from the vault recipe. Register it with AddSingleton:

services.AddOidcServices(options => { /* ... */ });
services.AddSingleton<IAuthServiceKeysProvider, MyProvider>();

The library adds its default provider only when none is registered, so your provider wins in either order. Registering it after AddOidcServices() simply reads most clearly.

Summary

Signing keys need three properties to work in production: they must be shared across all instances, they must survive restarts, and they must not live in source control. Abblix OIDC Server gives you three ways to meet them: a key custodian for a vault, OidcOptions.SigningKeys for a static key, and the IAuthServiceKeysProvider extension point for a store of your own.

Take the first recipe your environment allows. Keys in an HSM or KMS either never enter your process or rest only sealed by a vault key, and they rotate without a deploy. Static keys from a secret store handle small deployments with planned rotations. A Redis-backed provider gives you dynamic rotation without a deploy. A database-backed provider fits teams that already keep durable state in Postgres or SQL Server and accept holding the key there, encrypted at rest.

Whatever you pick, back the configuration with the multi-pod consistency test. It is the practical difference between a setup you can trust under rolling-deploy conditions and one whose failures only surface under production load.