# Persisting JWT Signing Keys in Production ## Overview Abblix OIDC Server signs JWT tokens using keys supplied through the `IAuthServiceKeysProvider` contract, empty by default. In any multi-instance deployment or across process restarts, this default is unsafe: unless the host provisions persistent, shared signing keys, instances will issue tokens that other instances or future restarts cannot validate. The result is intermittent `invalid_token` errors and mass logouts under rolling deploys. This guide covers the `IAuthServiceKeysProvider` contract, the three production failure modes caused by ephemeral or per-instance keys, three storage recipes (static secret, Redis-backed, database-backed), encrypting the stored key at rest, the key rotation discipline, and a verification test suitable for CI. ## Scope: signing keys, not Data Protection Before anything else, a disambiguation worth a minute of your time. ASP.NET Core Data Protection is the framework service that protects cookies, anti-forgery tokens, and `TempData`. Its key storage is a separate concern, configured via `services.AddDataProtection().PersistKeysTo...()`. If you've read that .NET teams routinely lose production Data Protection keys on restart and wonder whether that applies to Abblix, the answer is: it does not, because the library does not route signing through Data Protection. Abblix OIDC Server signing keys are the JSON Web Keys ([JWKs](https://www.abblix.com/en/docs/glossary-overview#jwk)) used to sign ID tokens, access tokens, and refresh tokens. They live behind a dedicated library contract, `IAuthServiceKeysProvider`. Both problems have the same shape (a key store that must survive restarts and be shared across pods) and both failure modes look identical from the outside (random authentication errors, phantom logouts). But the remedies are unrelated. Configuring Data Protection persistence does not persist your signing keys. You have to do both. The rest of this article is about the signing-key side; Data Protection returns only later, as a tool for encrypting that key at rest, never for its own cookie and anti-forgery duties. ## The library contract Abblix OIDC Server exposes signing-key access through a small interface: ```csharp public interface IAuthServiceKeysProvider { IAsyncEnumerable GetEncryptionKeys(bool includePrivateKeys = false); IAsyncEnumerable GetSigningKeys(bool includePrivateKeys = false); } ``` `IAsyncEnumerable` is deliberate. Retrieval may hit a cache, a [Redis](https://www.abblix.com/en/docs/glossary-overview#redis) hash, a database table, or an external [KMS](https://www.abblix.com/en/docs/glossary-overview#kms). The library does not assume synchronous in-memory access. The default implementation is `OidcOptionsKeysProvider`, which reads from `IOptions`: ```csharp public IReadOnlyCollection SigningKeys { get; set; } = []; public IReadOnlyCollection EncryptionKeys { get; set; } = []; ``` The default value of both properties is an empty collection. That is not a sensible production default, and it is not meant to be. The library ships an escape hatch (the options) and an extension point (the interface), and expects the host to choose which one to use before going live. The XML documentation on `OidcOptionsKeysProvider` says this out loud: > It is recommended to implement a dynamic resolution mechanism in production environments to enable seamless certificate replacement without the need for service reloading. ## Production failure modes Three concrete scenarios motivate the recipes in the rest of this guide. Each produces authentication errors that look random until you know the cause. ### Multi-pod key divergence A Kubernetes Deployment runs three pods of your auth-service. Each pod's startup code calls something like `JsonWebKeyFactory.CreateRsa()` to populate `OidcOptions.SigningKeys`. Each pod independently generates a different key pair. The three pods now have three different `jwks_uri` responses, three different sets of `kid` values, and three different private keys signing tokens that are all presented to the same load balancer. The user impact is intermittent 401 errors that correlate with nothing the user can control. Sticky sessions partially mask the problem; a rolling deploy exposes it. Your tracing shows nothing in application code. The only clue is that failures correlate with the `kid` claim on the token not matching the verifying pod's published [JWKS](https://www.abblix.com/en/docs/glossary-overview#jwks). Diagnosis is slow because the usual suspects (token expiry, clock skew, incorrect audience) all check out. ### Restart invalidates every token A single-pod deployment has keys generated at process start. Every [refresh token](https://www.abblix.com/en/docs/glossary-overview#refresh-token) in circulation is signed by the current process's in-memory private key. A rolling restart, a liveness-probe kill, or a Kubernetes eviction replaces the process. New keys, new `kid` values, and every refresh token that was valid a minute ago is now unrecognized. Users experience a mass logout. Mobile apps start retrying refresh, hit the auth endpoint harder than steady state, and some teams accidentally DoS themselves during a routine deploy. ### Keys in Git The third failure mode is avoided by engineers who know not to commit `.pem` files, but it keeps appearing in leaked-secrets audits anyway. A developer pastes a JWK into `appsettings.Development.json`, copies that file into `appsettings.Production.json` to "unblock staging", and a month later somebody greps the repo history and finds a live production private key in a commit. The code works. The deploy works. The security review three quarters later finds the key. ## Recipe A: static keys from a shared secret This is the simplest production-grade setup, appropriate when you have a small number of pods, a stable key set, and no strict rotation SLA. Keys live in a Kubernetes Secret (or Vault, or AWS Secrets Manager) and are mounted into every pod as an environment variable or file. All pods read the same keys at startup and never change them until a planned rotation. Generate a JWK once, store it in your secret manager, and load it through configuration: ```csharp services.AddOidcServices(options => { var jwkJson = Environment.GetEnvironmentVariable("ABBLIX_SIGNING_JWK") ?? throw new InvalidOperationException( "ABBLIX_SIGNING_JWK is required. See docs/signing-key-persistence."); var jwk = JsonSerializer.Deserialize(jwkJson) ?? throw new InvalidOperationException("ABBLIX_SIGNING_JWK is not valid JSON."); options.SigningKeys = new[] { jwk }; }); ``` The corresponding secret (Kubernetes example): ```bash kubectl create secret generic abblix-signing-key \ --from-literal=ABBLIX_SIGNING_JWK="$(cat signing-key.jwk.json)" \ --namespace=production ``` And the Deployment references it: ```yaml env: - name: ABBLIX_SIGNING_JWK valueFrom: secretKeyRef: name: abblix-signing-key key: ABBLIX_SIGNING_JWK ``` What this buys you: - Every pod sees the same keys at startup. No multi-pod divergence. - A pod restart reads the same secret and produces the same `jwks_uri`. Existing tokens remain valid. - The key never appears in git, in a built image, or in a runtime memory dump anyone can trivially grab. What this does not buy you: - Automatic rotation. A new key requires a manual secret update and a restart. - Graceful overlap during rotation. Unless you deliberately publish both the old and the new key for a window, clients will see JWKS churn and existing tokens may stop validating. For teams with low-volume auth traffic and rotations measured in months, Recipe A is often the right answer. For high-volume services with compliance-driven rotation policies, read on. :::note[Why not just put the JWK in appsettings.Production.json?] Two reasons. First, application settings files typically end up in your build artifact, Docker image, or deployment manifest, all of which are easier to accidentally leak than a secret manager entry. Second, `appsettings.*.json` is not designed for secrets and most organizations' scanning tooling does not treat it as such. Keep private keys in a dedicated secret store, always. ::: ## Recipe B: Redis-backed provider When rotation is frequent, or when you want the ability to swap keys without a deploy, a Redis-backed provider gives you a shared, out-of-process key store that every pod reads from on demand. The library deliberately does not depend on any particular Redis client, so the host owns the dependency. That keeps Abblix OIDC Server free of transitive weight for teams who use a different backend, and it keeps version conflicts from forcing your hand. Pseudocode (requires the host to have `StackExchange.Redis` available and an `IConnectionMultiplexer` registered): ```csharp public sealed class RedisAuthServiceKeysProvider( IConnectionMultiplexer redis, ILogger logger) : IAuthServiceKeysProvider { private const string SigningKeysHash = "abblix:signing-keys:active"; private const string EncryptionKeysHash = "abblix:encryption-keys:active"; public IAsyncEnumerable GetSigningKeys(bool includePrivateKeys = false) => ReadKeysAsync(SigningKeysHash, includePrivateKeys); public IAsyncEnumerable GetEncryptionKeys(bool includePrivateKeys = false) => ReadKeysAsync(EncryptionKeysHash, includePrivateKeys); private async IAsyncEnumerable ReadKeysAsync( string hashKey, bool includePrivateKeys) { var entries = await redis.GetDatabase().HashGetAllAsync(hashKey); foreach (var entry in entries) { if (entry.Value.IsNullOrEmpty) continue; JsonWebKey? jwk; try { jwk = JsonSerializer.Deserialize(entry.Value!); } catch (JsonException ex) { logger.LogWarning(ex, "Malformed JWK at {HashKey}/{Field}", hashKey, entry.Name); continue; } if (jwk is null) continue; yield return jwk.Sanitize(includePrivateKeys); } } } ``` Host registration: register your provider after `AddOidcServices()` so it overrides the library default: ```csharp services.AddOidcServices(options => { /* leave SigningKeys empty; Redis provider wins */ }); services.AddSingleton(); ``` What to put in Redis: - A hash per key-usage (one for signing, one for encryption). - Each field is a `kid`, each value is the full JWK serialized as JSON. - A separate small wrapper job (or an admin UI) adds new keys and retires old ones on schedule. Performance note: a production-quality Redis provider caches the result of `ReadKeysAsync` in-process for a short TTL (five to thirty seconds is typical). Reading Redis for every token-sign operation is unnecessary overhead, because keys rotate on human timescales, not per-request. Implement the cache with `IMemoryCache` or equivalent, and invalidate on a simple Redis pub/sub signal if you need sub-minute propagation. The library carries no Redis client dependency, so a Redis provider is host-owned code that pulls in `StackExchange.Redis` itself. The runnable sample in the repository is the database-backed recipe below, not this one; the Redis shape mirrors it, swapping the table query for a hash read. ## Recipe C: database-backed provider The library does not prescribe an ORM, and neither does this recipe. If you already have EF Core, use EF Core. If you prefer Dapper, use Dapper. If you're writing raw ADO.NET against a legacy stored-procedure layer, that works too. What matters is the table shape and the query. Suggested table: ```sql CREATE TABLE abblix_signing_keys ( kid TEXT PRIMARY KEY, key_usage TEXT NOT NULL, -- 'sig' or 'enc' algorithm TEXT NOT NULL, -- e.g. 'RS256', 'ES384' jwk_json TEXT NOT NULL, -- full JWK, including private material not_before TIMESTAMPTZ, -- NULL means effective immediately not_after TIMESTAMPTZ, -- NULL means indefinite is_active BOOLEAN NOT NULL DEFAULT TRUE, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX abblix_signing_keys_active ON abblix_signing_keys (key_usage, not_after) WHERE is_active = TRUE; ``` The effective query for `GetSigningKeys`: ```sql SELECT jwk_json FROM abblix_signing_keys WHERE key_usage = 'sig' AND is_active = TRUE AND (not_before IS NULL OR not_before <= NOW()) AND (not_after IS NULL OR not_after > NOW()) ORDER BY created_at DESC; ``` Wrap that in whichever data-access technology you already use. Cache the result in-process with the same short TTL you'd use for the Redis recipe. Implement `IAuthServiceKeysProvider` on top of the query-plus-cache, register it in [DI](https://www.abblix.com/en/docs/glossary-overview#di), done. Private keys at rest deserve their own conversation, and the honest answer is that an application database is not their real home. In a high-assurance system signing keys live in an [HSM](https://www.abblix.com/en/docs/glossary-overview#hsm) or a managed KMS (AWS KMS, Azure Key Vault, GCP KMS, HashiCorp Vault), where the private half ideally never leaves the boundary in plaintext. This library imposes a ceiling here worth stating plainly: `IAuthServiceKeysProvider` hands a `JsonWebKey` with private material to an in-process signer, so vault-resident signing (the key never leaving the HSM) is not reachable through that seam. What the keys provider can do is fetch the private material from a KMS per use, which brings the plaintext private half into process memory to sign. The database-backed recipe here is a documented, deliberate tradeoff for teams that accept DB key custody. If you take it, encrypt the key at rest at a minimum, either at the database layer (pgcrypto, Always Encrypted, TDE with additional protections) or at the application layer with ASP.NET Data Protection (see [Encrypting the stored key at rest](#encrypting-the-stored-key-at-rest)). Whichever layer you pick, the encryption relocates key custody to that layer's own master key rather than removing it, as the next section traces for Data Protection. Prefer keeping only key IDs in the database while fetching private material from a KMS per use. Left unencrypted and unrotated, a database holding raw RSA private keys is worse than a Kubernetes Secret. A runnable version of this recipe, backed by EF Core and SQLite, lives in the `AspNetIdentitySample` project of the [Getting Started repository](https://github.com/Abblix/Oidc.Server.GettingStarted). It generates and persists a signing key on first run, encrypts it at rest with ASP.NET Data Protection (see the next section), and reloads the same key on every restart, so tokens issued before a restart keep validating. The residual custody concerns are marked with `#warning` directives in the sample, so a developer copying it toward production trips over them at build time. ## Encrypting the stored key at rest Storing a private key as plaintext in any database is the weakest form of this recipe. One step up, without reaching for a KMS, is to encrypt the JWK before it is written and decrypt it on read. ASP.NET Core Data Protection is the built-in tool for that: create a protector with a fixed purpose string, `Protect` the serialized JWK on the way in, `Unprotect` it on the way out. ```csharp // on write (seeding or rotation) var protector = dataProtectionProvider.CreateProtector("MyApp.SigningKeys.v1"); record.JwkJson = protector.Protect(JsonSerializer.Serialize(key)); // on read, inside your IAuthServiceKeysProvider var json = protector.Unprotect(record.JwkJson); var key = JsonSerializer.Deserialize(json); ``` The `AspNetIdentitySample` sample does this, so a raw dump of the SQLite file yields ciphertext, not a signing key. Be precise, though, about what this buys and what it does not. There is an irony worth naming. The [scope note](#scope-signing-keys-not-data-protection) at the top of this article warned that Data Protection keys and signing keys are two separate stores with the same failure modes. Encrypting your signing key with Data Protection does not merge them, it stacks them: your signing key is now exactly as durable and as shared as the Data Protection key ring that encrypts it. The custody problem moves one layer down, onto a key ring that must be persisted and shared exactly as this article demands of the signing key, and protected on top of that: a smaller, better-understood surface to defend, but not one you can ignore. - **Persisted.** With no explicit configuration the key ring lands in a per-user profile folder on a developer machine, which is exactly why a local restart still decrypts the signing key. Inside a container with no user profile that fallback is not available, and Data Protection degrades to ephemeral in-memory keys lost on the next restart. Persist and back up the ring, or a lost ring leaves every stored signing key unrecoverable. Call `PersistKeysToFileSystem(...)` a durable location (or a shared store) and `SetApplicationName(...)` so every instance derives the same keys, rather than relying on the developer-machine default. - **Shared.** Two instances with two different key rings cannot read each other's encrypted signing keys, so an unshared ring has the same multi-instance problem this article opened with, one layer down. Persist the ring to a store every instance reads, using the backend-specific extension: `PersistKeysToFileSystem` for a shared volume (in-box), `PersistKeysToStackExchangeRedis`, `PersistKeysToAzureBlobStorage`, or `PersistKeysToDbContext`, each in its own package. There is no generic `PersistKeysTo`. - **Protected.** A persisted ring is itself plaintext at rest unless you protect it with `ProtectKeysWithCertificate(...)` or a cloud KMS (`ProtectKeysWithAzureKeyVault(...)`). Without that, the Data Protection keys sit next to the database you were protecting, and the encryption defends only against a leaked database file, not against read access to the host filesystem. The certificate option only defers custody again (the cert's own private key must be loadable to open the ring); a cloud KMS is what actually terminates the chain. Concrete recommendations, by deployment shape: - **Single instance, development or low assurance.** Data Protection with `PersistKeysTo` a durable path is enough. Know that without `ProtectKeysWith`, you are defended against a leaked database backup but not against an attacker who can read the host filesystem. - **Multiple instances.** The key ring must be both shared (a store every instance reads: a mounted volume, a blob container, Redis, a database table) and protected (a certificate or KMS). Configure both: miss the shared store and validation fails across pods, miss the protector and the encryption adds nothing over the database's own protection. - **High assurance or compliance-bound.** Do not keep private key material in your database at all, encrypted or not. Put the key in an HSM or a managed KMS and store only key IDs in the database. Mind the library ceiling from Recipe C: `IAuthServiceKeysProvider` can fetch and unwrap the private key from the vault per use (it then signs in process), but a key that never leaves the HSM would require replacing the token-signing pipeline, not just the keys provider. This is the real home for signing keys; Data Protection encryption is a pragmatic middle ground, not a substitute. ## Key rotation discipline Whichever storage you pick, rotation needs the same four moves. - **Generate a new key ahead of time.** Give it a unique `kid`, a `not_before` set to about ten minutes from now, and insert it as `is_active = TRUE`. Your JWKS endpoint will start publishing the new key immediately, but the signing pipeline will keep using the previous one until the `not_before` window opens. - **Start signing with the new key.** At the configured `not_before`, your signing code should prefer the newest active key. Relying parties who fetched JWKS in the last ten minutes already have the new `kid` in their cache. Freshly issued tokens validate. - **Retain the old key for the token lifetime plus a buffer.** Set the old key's `not_after` to now plus your longest refresh-token lifetime plus a safety margin of at least an hour. During this overlap window, both keys are published in JWKS, old tokens validate against the old key, and new tokens validate against the new key. Nobody is locked out. - **Retire the old key.** Once no extant token could still be signed by it, set `is_active = FALSE` and drop it from JWKS. Keep the row around for an audit trail, but stop publishing the public half. A common beginner mistake is to rotate the key and immediately remove the old one, which invalidates every outstanding refresh token. Another is to skip `not_before` and start signing with a key the JWKS endpoint hasn't published yet, which causes a brief window of 401s for every client that cached JWKS too recently. The overlap window is not a nicety, it is the whole point. :::note[How long should the overlap window be?] At minimum, the longest refresh-token lifetime you issue. If your refresh tokens live for thirty days, the old key must remain verifiable for thirty days after you stop signing with it. For most deployments, forty-five days is a reasonable default. For short-lived tokens in higher-risk environments, shorter overlap windows reduce the blast radius if a key is compromised. ::: ## A multi-pod consistency test worth writing An integration test you can run in CI, using ephemeral keys but a shared store: - Start two auth-service instances pointed at the same Redis or database. - Issue a token from instance A. - Present that token to instance B's [token introspection](https://www.abblix.com/en/docs/glossary-overview#token-introspection) endpoint. - Assert the introspection result is `active: true`. Repeat with a forced key rotation between issue and validate. Assert the old-signed token still validates during the overlap window. Assert it fails after the window elapses. This test catches almost every real configuration mistake in this area before it reaches production. It runs in CI alongside normal integration tests. ## Registering your provider To override the library's default `IAuthServiceKeysProvider`, register your implementation after `AddOidcServices()`: ```csharp services.AddOidcServices(options => { /* ... */ }); services.AddSingleton(); ``` Order matters. Microsoft's default DI container returns the most recently registered implementation when resolving a single service, so your `AddSingleton` must come after `AddOidcServices()` to win. Registering in the opposite order silently leaves the library default in place. ## Summary Signing keys need three properties to work in production: they must be shared across all instances, they must survive restarts, and they must not live in source control. Abblix OIDC Server provides the `IAuthServiceKeysProvider` extension point exactly so you can meet all three without fighting the library. Pick the simplest recipe that meets your rotation needs. Static keys from a secret store handle small deployments with planned rotations. A Redis-backed provider gives you dynamic rotation without a deploy. A database-backed provider fits naturally in teams that already keep durable state in Postgres or SQL Server. Whatever you pick, back the configuration with the multi-pod consistency test. It is the practical difference between a setup you can trust under rolling-deploy conditions and one whose failures only surface under production load.