# Multi-Tenancy in Abblix OIDC Server

A software vendor signs in the users of two customers, Acme and Globex, from one deployment at `auth.example.com`. Acme's application posts a token request to `https://auth.example.com/tenants/acme/connect/token`. The server reads the host name and the path, finds the tenant whose [issuer](https://www.abblix.com/en/docs/glossary-overview#issuer) is `https://auth.example.com/tenants/acme`, and from then on, everything the server holds for that request is Acme's: the client that authenticates, the key that signs, the issuer written into the token. The users and the consents come from your own providers and are not separated for you, as the caution in [Serve two tenants](#turn-it-on) explains. Globex's tokens name a different issuer and are signed with a different key.

That is how a server with several tenants finds the tenant of a request. Each tenant is an issuer of its own, reached at its own address, and a server that serves one issuer declares one tenant. What the server keeps in its own stores and key rings is kept per tenant. The users, the consents and the client registrations a host stores itself are yours to separate, as the sections below describe.

## Serve two tenants {#turn-it-on}

Serving the tenants takes two calls. `AddOidcServices` registers the server and takes the tenants as options; `UseMultiTenancy` resolves each request to its tenant in the request pipeline. The sample uses the Minimal API package (`Abblix.OIDC.Server.MinimalAPI`). An MVC host references `Abblix.OIDC.Server.MVC` instead, imports `Abblix.Oidc.Server.Mvc` in place of `Abblix.Oidc.Server.MinimalApi`, and calls `app.MapOidcControllers()` where this sample calls `app.MapOidcEndpoints()`; the tenant calls are the same. Here is a complete `Program.cs` serving Acme and Globex:

```csharp
using System.Security.Cryptography;
using System.Text;
using Abblix.Jwt;
using Abblix.Oidc;
using Abblix.Oidc.Server.AspNetCore.MultiTenancy;
using Abblix.Oidc.Server.Features.ClientInformation;
using Abblix.Oidc.Server.Features.Consents;
using Abblix.Oidc.Server.Features.MultiTenancy;
using Abblix.Oidc.Server.MinimalApi;

var builder = WebApplication.CreateBuilder(args);

var licenseJwt = builder.Configuration["Oidc:LicenseJwt"];
var acmeSecret = builder.Configuration["Tenants:AcmeSecret"]!;
var globexSecret = builder.Configuration["Tenants:GlobexSecret"]!;

// What every Abblix OIDC Server host registers
builder.Services.AddAuthentication().AddCookie();
builder.Services.AddAuthorization();
builder.Services.AddCors();
builder.Services.AddMemoryCache();
builder.Services.AddDistributedMemoryCache();
builder.Services.AddHttpContextAccessor();

// Grants every client everything it asks for, which suits only clients of your own;
// to ask the end user for consent, register your own IUserConsentsProvider instead
builder.Services.AddConsentGrantedForEveryClient();

builder.Services.AddOidcServices(options =>
{
    options.LicenseJwt = licenseJwt;
    options.Tenants.Add(Tenant("acme", "/tenants/acme", "acme-client", acmeSecret));
    options.Tenants.Add(Tenant("globex", "/tenants/globex", "globex-client", globexSecret));
});

var app = builder.Build();
app.UseMultiTenancy();
app.UseCors();
app.UseAuthentication();
app.UseAuthorization();
app.MapOidcEndpoints();
app.Run();

static TenantDefinition Tenant(string id, string path, string clientId, string secret) => new()
{
    Id = id,
    Issuer = new Uri("https://auth.example.com" + path),
    Clients =
    [
        new ClientInfo(clientId)
        {
            ClientSecrets = [new ClientSecret { Sha512Hash = SHA512.HashData(Encoding.UTF8.GetBytes(secret)) }],
            TokenEndpointAuthMethod = ClientAuthenticationMethods.ClientSecretPost,
            AllowedGrantTypes = [GrantTypes.ClientCredentials],
        },
    ],
    SigningKeys = [JsonWebKeyFactory.CreateRsa(PublicKeyUsages.Signature)],
};
```

The registrations before `AddOidcServices` are the ones every host needs, and the [getting started guide](https://www.abblix.com/en/docs/getting-started-guide) explains each. Without the distributed cache the server does not start, and neither does it without a consent provider: `AddConsentGrantedForEveryClient` grants every request, which a demo with clients of your own can afford, and a host with clients it does not control registers its own `IUserConsentsProvider`. The sample registers no `IUserInfoProvider`, which `client_credentials` does not need: the server then knows no users, issues no ID token, answers userinfo with `invalid_token`, and logs a warning that the provider is missing each time something asks about a user. A host that signs users in registers its own, as the caution below shows. The signing keys here are generated in the process, which suits a demo: every restart and every instance would get different ones. A real deployment keeps each tenant's keys somewhere durable, as [signing key persistence](https://www.abblix.com/en/docs/signing-key-persistence) describes, or lets a custodian hold them.

`UseMultiTenancy` adds routing itself, right after the resolution, because a tenant served under a path moves that path into the path base and the routes must be matched against what remains. So it goes before any `UseRouting` of your own, and a call placed after one throws at startup. If it is missing from the pipeline, the server stops as soon as it starts listening, and the error names the missing call and why this server needs it. Two hosts can leave it out. One finds the tenant of a request through a tenant accessor of its own. The other serves the one tenant its settings declare, at an issuer without a path, and registers no tenant store, catalog or store writer of its own: it serves that tenant at whatever host name a request arrives with, except under a path base the host sets, and the [getting started guide](https://www.abblix.com/en/docs/getting-started-guide) builds such a host. `UseMultiTenancy` goes before `UseAuthentication` too, so the authentication cookie takes the tenant's path.

:::note[A second tenant needs a license]
The free tier serves one issuer, and each tenant is an issuer, so serving a second tenant takes a license that allows several issuers; the [pricing page](https://www.abblix.com/en/oidc-server/pricing) says how to get one. Without it the sample does not start: the server refuses to start when the tenants declared in the options outnumber the issuers the license allows, or when one of them is at an issuer the license does not name. The [licensing section](#license) describes the rest of what the license decides.
:::

:::caution[Users and sign-in steps are shared]
The server keeps no users of its own, and it asks the same providers for every tenant: `IUserInfoProvider`, `IUserConsentsProvider`, and `IUserInteractionRequirement` for the steps a user must complete before anything is issued. Each of them is handed the tenant of the request as its first argument, so a host whose tenants have separate users looks the user up in that tenant's directory:

```csharp
using System.Text.Json.Nodes;
using Abblix.Oidc.Server.Features.MultiTenancy;
using Abblix.Oidc.Server.Features.UserAuthentication;
using Abblix.Oidc.Server.Features.UserInfo;

internal sealed class TenantUsers(IUserDirectory directory) : IUserInfoProvider
{
    public Task<JsonObject?> GetUserInfoAsync(
        TenantDefinition tenant, AuthSession authSession, IEnumerable<string> requestedClaims)
        => directory.FindAsync(tenant.Id, authSession.Subject, requestedClaims);
}
```

`IUserDirectory` stands for your own user store. A host whose tenants share their users can ignore the tenant in its user lookup, but consents need the tenant even then: a consent recorded without it counts at every tenant that serves a client under the same id, so keep each consent under the tenant of the request and answer only with the consents given there.

The login page is yours too, and it checks the user against the directory of the tenant the request came to. The sign-in session belongs to the tenant it was signed in at: the server keeps the tenant's id and generation in the authentication cookie, and reads a session of another tenant, or one without that mark, as no session, so its user signs in again. A request outside every tenant, such as a page of your application outside their paths, sees no session, and a sign-in there throws an `InvalidOperationException`. A cookie the browser carries to another tenant, through a cookie domain several tenant hosts share or a widened cookie path, is written over by a sign-in there without ending the first session, and the first tenant's clients are not told; keep each tenant's cookie to its own host and path.
:::

### Running it locally {#running-it-locally}

An issuer must be an https address. `OidcOptions.AllowHttpIssuer = true` lets http issuers through for local development, for every tenant at once, and a tenant whose issuer is an http address is then served over http. The commands below keep https, with the ASP.NET Core development certificate (`dotnet dev-certs https`), so the demo runs as a deployment would.

Scheme and port are not compared, so the tenants above answer at any port as long as the request's host name is `auth.example.com`. On a developer machine, point that name at your own: add `127.0.0.1 auth.example.com` to the hosts file, or pass `--resolve` to curl.

The sample reads the two client secrets and the license from configuration. Keep them in user secrets, which the host reads in the Development environment, and start the host on the address the commands below use. The arguments after `--` go to the application; before it, `dotnet run` would read `--environment` as an environment variable of its own:

```bash
dotnet user-secrets init
dotnet user-secrets set "Tenants:AcmeSecret" "$ACME_SECRET"
dotnet user-secrets set "Tenants:GlobexSecret" "$GLOBEX_SECRET"
dotnet user-secrets set "Oidc:LicenseJwt" "$LICENSE_JWT"
dotnet run -- --urls https://localhost:5443 --environment Development
```

The host name matters most when your license names its issuers, because then the issuers must stay exactly as licensed. With the host listening on `https://localhost:5443`, the discovery document and the token endpoint of Acme:

```bash
curl -k --resolve auth.example.com:5443:127.0.0.1 \
  https://auth.example.com:5443/tenants/acme/.well-known/openid-configuration

curl -k --resolve auth.example.com:5443:127.0.0.1 \
  -X POST https://auth.example.com:5443/tenants/acme/connect/token \
  -d grant_type=client_credentials \
  -d client_id=acme-client \
  -d client_secret=$ACME_SECRET
```

The access token that comes back names `https://auth.example.com/tenants/acme` as its `iss`. The same two requests under `/tenants/globex` return Globex's document and a token with Globex's issuer, signed with Globex's key. Acme's client sent to Globex's token endpoint is refused with `invalid_client`.

## How a request finds its tenant {#how-a-request-finds-its-tenant}

A request belongs to the tenant whose issuer names the request's host name and the longest prefix of its path that ends at a segment boundary. For the issuer `https://auth.example.com/tenants/acme`, a request to `/tenants/acme/connect/token` reaches the token endpoint as `/connect/token`: the issuer's path becomes the request's path base, so the endpoints are routed on the path that remains, and every address built from the path base lies under the issuer. That covers the endpoints discovery advertises, the cookie path, and the address a proof or an assertion is checked against.

An issuer can be a whole host, such as `https://acme.example.com`, or a path on a shared one, as above. Scheme and port are not compared, because behind a proxy the request carries its own; your application's forwarded-headers setup decides those.

A request that no issuer covers passes through without a tenant, since your application may serve other things on the same host. The OpenID endpoints answer such a request with 404, so a path under `/tenants/` that names no tenant is a 404 too.

### The RFC 8414 address {#rfc-8414-address}

The form [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414#section-3.1) defines for an issuer with a path, the well-known string inserted between the host and the path, reaches the same tenant: `/.well-known/oauth-authorization-server/tenants/acme` serves the metadata of the tenant at `/tenants/acme`. It is read only at the root of the host, where RFC 8414 puts it, so it is not read when your application mounts the server under a path base of its own, and only when the path after the well-known string is exactly that issuer's. The server routes this form itself, and the [discovery article](https://www.abblix.com/en/docs/discovery-and-well-known-endpoints) explains why the two discovery rules pull apart once an issuer has a path.

### Where tenants under paths fall short {#paths-share-a-host}

Tenants served under paths of one host share what the browser ties to the host:

- One origin. Script running on one tenant's pages can read another tenant's pages on the same host. When tenants are separate customers, give each a host of its own.
- No nesting. The outer tenant's cookie, scoped to its path, would reach a tenant under it, so the server refuses a tenant whose issuer's path lies under another tenant's on the same host: the root `https://auth.example.com` next to `/tenants/acme`, or `/tenants/acme` next to `/tenants/acme/eu`. Whole path segments are compared, case-sensitively, so `/tenants/acme2` next to `/tenants/acme` is accepted.

### The host and the proxy {#host-and-proxy}

Tenant selection trusts the host name the pipeline presents, so:

- Restrict `AllowedHosts` to the host names your tenants use.
- Accept forwarded headers only from your own proxy.
- Refuse plain http at the edge.

## What each tenant declares {#what-each-tenant-declares}

A tenant is a `TenantDefinition`. `Id` and `Issuer` are always required, and while the keys come from each tenant's settings, so is at least one key in `SigningKeys`. Everything that belongs to one issuer is a member of the definition, never of `OidcOptions`:

- The clients. A client is known only at the tenant that lists it, or at which it was registered dynamically, so two tenants may each have a client under the same id.
- The scopes and resources the tenant defines, and the resource a token is issued for when a request names none.
- The pages a user is sent to: sign-in, registration, account selection, consent, and required interaction. A relative address is a page under the tenant's issuer.
- The security profile every client of the tenant is held to at minimum.
- The pairwise subject settings. Each tenant keeps its own salt, so the pseudonyms two tenants give one user cannot be matched to each other.
- The keys it signs with and decrypts with, under the rule after this list.
- The base address of the tenant's mutual-TLS endpoint aliases, which carries no path of its own because the aliases keep the issuer's path.
- The generation, which says which creation of the tenant under its id this is. The manager sets it, as [changing tenants](#changing-tenants-while-the-server-runs) explains, so a definition you write usually leaves it empty.

The id, and the generation when the tenant has one, may hold only ASCII letters, digits, `-` and `_`, because the names the server keeps a tenant's data and keys under are built from them. A tenant with any other character is refused wherever it enters: at startup, at a reading of the store, which leaves it out and logs it, and by the manager. No check a host configures lets it through.

How the server keeps keys is one choice for every tenant:

- The keys come from each tenant's own settings, `SigningKeys` and `EncryptionKeys`.
- A custodian holds them (`CustodianKeys`). Every tenant names keys of its own, and no two tenants name the same one; a tenant whose `CustodianKeys.EncryptionAlgorithm` names a key management algorithm the host has not registered is refused at startup.
- The server mints them, seals them to a key-encryption key a custodian holds, and keeps each tenant's ring in a shared ring store under the tenant's id and generation. A tenant then declares no keys.
- A key provider of your own (`IAuthServiceKeysProvider`) answers for each tenant's keys, given the tenant, and the server checks none of what the tenants declare, unless a custodian holds the keys behind it: then the custodian checks above still apply.

With the first three, a tenant that declares keys the choice does not read is refused.

The server also checks the list of tenants itself. It refuses a tenant with no id, an issuer that is not an https address without a query, fragment, user name or password, an issuer with an empty segment in its path, two tenants with the same id, two tenants served at the same host and path, a tenant whose issuer's path lies under another tenant's on the same host, and, while the keys come from each tenant's settings, a tenant with no `SigningKeys` and a key that two tenants declare, compared by its thumbprint in `SigningKeys` and `EncryptionKeys` alike, which refuses both tenants, and, while the keys are held by a custodian, a tenant with no `CustodianKeys` and a key name that two tenants give, which refuses both as well. [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414#section-2) makes the issuer an https address with no query or fragment; http passes only with `AllowHttpIssuer`, as [running it locally](#running-it-locally) describes. A tenant declared in the options that fails a check stops the server from starting. A tenant read from a store that fails a check is left out and logged, both of a nested pair or of a shared key, and the others are served.

## What keeps one tenant's tokens from another {#what-keeps-tokens-apart}

The server accepts back only a token whose `iss` is the issuer of the tenant the request reached. A refresh token, an introspection or revocation request, or a userinfo call carrying another tenant's token is answered as for an unknown token: introspected at Globex, a token Acme issued is `{"active":false}`. Introspection is an optional endpoint: call `builder.Services.AddIntrospection()` (namespace `Abblix.Oidc.Server.Features`). Separate keys are the second wall, and the server refuses two tenants that declare the same key, in their settings or at a custodian, as the [checks above](#what-each-tenant-declares) list; a key provider of your own answers for that itself. A resource server that trusts several tenants must check `iss` itself, against the issuer whose keys verified the token.

## What the server keeps apart for you {#what-the-server-keeps-apart}

Beyond the registries built from each definition, the server keeps the data each tenant owns apart. The entity storage holds the authorization codes, pushed requests, the requests clients poll in device authorization and back-channel authentication, token statuses, session registries, rate-limit counters, and the nonce secret, and all of it is kept per tenant. The replay cache is partitioned the same way, and so are the per-caller limits for client authentication failures, introspection, and revocation.

Because those limits are per tenant, someone who tries the same client credentials at every tenant gets each tenant's allowance, so the attempts allowed grow with the number of tenants. Limits on the overall load belong to your proxy.

A host that brings its own storage or replay cache registers it under the key `TenantServiceKeys.Shared`, and the server keeps each tenant's entries in it under keys of the tenant's own. The registration may come before or after `AddOidcServices`. The server builds every key it writes to the entity storage from the tenant, so a storage of your own needs no notion of tenants. A replay cache registered without the key after `AddOidcServices` would take the place of the cache the server keeps apart per tenant, and answer every tenant from one set, so startup refuses it and names the service. A rate limiter of your own spends one tenant's budget: register it under `TenantServiceKeys.PerTenant` with the budget's name, as a factory or a type that builds a new limiter each time it is resolved, because each tenant resolves its own.

## Keeping tenants in a store {#keeping-tenants-in-a-store}

Tenants declared in `OidcOptions.Tenants` change only with a restart. A deployment whose tenants come and go keeps them in a store instead, and the server reads the store at startup and again every `OidcOptions.RefreshTenantsEvery`, a minute by default, which this article calls the refresh period. A tenant the store gains, changes, or loses therefore reaches every instance within one period, provided its readings succeed. Requests are answered from the last successful reading, never from the store itself.

A store that cannot be read at startup stops the process, since nothing is being served yet. After startup a failed reading is logged and the instance keeps the tenants of the last good one, because taking the process down would turn an outage of the store into an outage of every tenant; a change waits for the next reading that succeeds.

The store is `ITenantStore`, one method that lists every tenant as a `StoredTenant(TenantDefinition Tenant, string Version)`. The version changes whenever the definition does, and only then; while it stays the same, the server keeps serving what it already built from the definition.

Register your store as a singleton, and it takes the place of the default one, which serves `OidcOptions.Tenants`. With your own store registered, the server serves only what the store lists: tenants added to `OidcOptions.Tenants` are checked at startup but neither served nor handed to `ITenantOpening`, so seed the store instead. A store of your own is also the only way to start with no tenant at all; a server that takes its tenants from its settings and declares none refuses to start. If the host changes tenants through the manager described below, the store also implements `ITenantStoreWriter`, registered as a singleton too. A writer registered without a store of your own stops the server at startup, since `OidcOptions.Tenants` never holds what it writes, and the error names the writer. This in-memory store, taken from the end-to-end suite, does both:

```csharp
using System.Collections.Concurrent;
using Abblix.Oidc.Server.Features.MultiTenancy;

internal sealed class MemoryTenantStore : ITenantStore, ITenantStoreWriter
{
    private readonly ConcurrentDictionary<string, StoredTenant> _tenants = new(StringComparer.Ordinal);

    public Task<IReadOnlyCollection<StoredTenant>> ListAsync(CancellationToken cancellationToken)
        => Task.FromResult<IReadOnlyCollection<StoredTenant>>(_tenants.Values.ToArray());

    public Task<string?> AddAsync(TenantDefinition tenant, CancellationToken cancellationToken)
    {
        var stored = new StoredTenant(tenant, NewVersion());
        return Task.FromResult(_tenants.TryAdd(tenant.Id, stored) ? stored.Version : null);
    }

    public Task<string?> UpdateAsync(
        TenantDefinition tenant,
        string expectedVersion,
        CancellationToken cancellationToken)
    {
        if (!_tenants.TryGetValue(tenant.Id, out var current) || current.Version != expectedVersion)
            return Task.FromResult<string?>(null);

        var stored = new StoredTenant(tenant, NewVersion());
        return Task.FromResult(_tenants.TryUpdate(tenant.Id, stored, current) ? stored.Version : null);
    }

    public Task<bool> RemoveAsync(string tenantId, string expectedVersion, CancellationToken cancellationToken)
        => Task.FromResult(
            _tenants.TryGetValue(tenantId, out var current)
            && current.Version == expectedVersion
            && _tenants.TryRemove(new KeyValuePair<string, StoredTenant>(tenantId, current)));

    private static string NewVersion() => Guid.NewGuid().ToString("N");
}
```

One instance serves both interfaces, so register that instance under each, and declare no tenants in the `AddOidcServices` call of the first sample:

```csharp
var store = new MemoryTenantStore();
builder.Services.AddSingleton<ITenantStore>(store);
builder.Services.AddSingleton<ITenantStoreWriter>(store);
builder.Services.AddOidcServices(options => options.LicenseJwt = licenseJwt);
```

A store starts empty, and so does the server reading it. Create the first tenants through the manager once the application runs, as the next section shows, or seed the store before it starts. A tenant seeded directly carries the generation you give it, so give it one:

```csharp
var acme = Tenant("acme", "/tenants/acme", "acme-client", acmeSecret);
await store.AddAsync(
    new TenantDefinition
    {
        Id = acme.Id,
        Issuer = acme.Issuer,
        Clients = acme.Clients,
        SigningKeys = acme.SigningKeys,
        Generation = Guid.NewGuid().ToString("N"),
    },
    CancellationToken.None);
```

The contract worth copying is in the return values:

- `AddAsync` returns the new version, or null when a tenant is stored under the id already.
- `UpdateAsync` and `RemoveAsync` name the version the caller read and apply only while the store still holds it, as one conditional write, so a change made meanwhile by another instance is never overwritten unseen. A database store makes the same decision in one conditional statement rather than a read followed by a write.
- The server compares versions for equality only, so any value works as long as every write produces a new one.

A definition holds secrets: a tenant's `SigningKeys` serialize with their private half, so a store that keeps definitions has to protect them as it would any private key. Check with a test of your own that the serialization round-trips the whole definition, client secrets and keys included.

:::caution[A listing that leaves a tenant out removes it]
The server reads a tenant missing from a listing as removed. Within two refresh periods it releases the tenant and deletes the keys it minted for it, and those cannot be recovered. So `ListAsync` returns the whole list or fails; it never returns a partial or an empty list because a query timed out. If you want an undo, keep removed tenants in your store under a mark of your own and leave them out of the listing only when you are sure.
:::

### The generation {#the-generation}

The generation is optional, and what it gives depends on who assigns it:

- A tenant created through the manager gets its generation from the manager.
- A store the host writes by other means assigns it itself, from ASCII letters, digits, `-` and `_` only (a GUID in the `"N"` format qualifies). A generation with any other character leaves the tenant out and logged.
- A store that assigns none serves the tenant without a generation. A tenant created again under that id then inherits what the old one stored.
- A tenant declared in `OidcOptions.Tenants` has no generation either, with the same inheritance. Moving such a tenant into a store that assigns a generation starts it empty.

A store must keep the generation exactly as given and never reuse one. The next section explains what the generation keeps apart.

## Changing tenants while the server runs {#changing-tenants-while-the-server-runs}

`ITenantManager` creates, changes, and removes tenants on a running server. It refuses a change that the startup checks would refuse, and it writes through your `ITenantStoreWriter`. Each instance checks a change against what the store lists at that moment and makes its own changes one at a time, so two instances changing tenants at once can store a pair the next reading refuses, such as two tenants at one address; that reading leaves them out and logs them. A tenant whose `ITenantOpening` fails is also left out after it was stored, and offered again at the next reading. The server registers the manager itself; a host without a writer gets an `InvalidOperationException` from any of the manager's methods.

Each method returns a result: the tenant as stored, or a `TenantChangeRefusal` whose `Reason` says what happened and whose `Message` says what is wrong:

- `Invalid`: the tenants as the change would leave them fail a startup check.
- `AlreadyExists` and `NotFound`: the store holds a tenant under the id already, or holds none.
- `Conflict`: the tenant was changed since it was read, so re-read it and decide again.
- `BeyondLicense`: a new tenant is refused when the store already holds as many tenants as the license allows issuers. The manager counts only what the store holds; the [licensing section](#license) says what that means for a full license.
- `IssuerNotLicensed`: the license names its issuers, and the tenant would be created at, or moved to, an issuer it does not name. A change that keeps the tenant's issuer passes even after the license narrowed.

`UpdateAsync` replaces the definition as a whole, keeping the stored generation, so send the tenant as it should be after the change, clients and keys included. `TenantDefinition` is a class rather than a record, so build the new definition from the old one's values. In a Minimal API host the manager is easiest to reach by injecting it into an endpoint of your own:

```csharp
// Administration endpoints of your own, placed before app.Run(); put them behind your own authorization.
// The manager and the store come from the container, injected into each handler.
app.MapPost("/admin/tenants/{id}", async (
    string id, ITenantManager manager, CancellationToken cancellationToken) =>
{
    // The client secret is made here and returned once; the tenant keeps only its hash
    var secret = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32));
    var created = await manager.CreateAsync(
        Tenant(id, "/tenants/" + id, id + "-client", secret),
        cancellationToken);

    // refusal.Reason says what to do about it, refusal.Message what is wrong
    return created.Match<IResult>(
        stored => Results.Ok(new { stored.Version, stored.Tenant.Generation, ClientSecret = secret }),
        refusal => Results.Conflict(new { Reason = refusal.Reason.ToString(), refusal.Message }));
});

app.MapPost("/admin/tenants/{id}/keys", async (
    string id, ITenantManager manager, ITenantStore store, CancellationToken cancellationToken) =>
{
    // Read the version from the store's listing and send the tenant as it should be
    var stored = (await store.ListAsync(cancellationToken)).Single(t => t.Tenant.Id == id);
    var changed = await manager.UpdateAsync(
        new TenantDefinition
        {
            Id = stored.Tenant.Id,
            Issuer = stored.Tenant.Issuer,
            Clients = stored.Tenant.Clients,
            SigningKeys = [.. stored.Tenant.SigningKeys, JsonWebKeyFactory.CreateRsa(PublicKeyUsages.Signature)],
        },
        stored.Version,
        cancellationToken);

    return changed.Match<IResult>(
        updated => Results.Ok(new { updated.Version, updated.Tenant.Generation }),
        refusal => Results.Conflict(new { Reason = refusal.Reason.ToString(), refusal.Message }));
});
```

After a change is written, the instance that made it reads the store again and serves the change on the next request, with no restart and no wait for the next refresh. Other instances serve it at their next reading. If that reading fails, or a created tenant cannot be readied yet, this instance serves the change from its next reading that succeeds, and the call still reports the change as made, because it is written.

Every tenant created through the manager, or by a store that assigns one, gets a generation when it is created, and keeps it for as long as it lives. What the server keeps for a tenant, including its stored codes and tokens, its minted keys, and what it builds from the definition, is kept under the id and the generation together. A tenant with a generation that is created again under the id of a removed one is a different creation and starts with none of it, which is what keeps a returning customer from inheriting the old tenant's sessions. So wherever an id may be used again, create tenants through the manager or a store that assigns generations.

## When a tenant arrives and leaves {#when-a-tenant-arrives-and-leaves}

`ITenantOpening` runs when a tenant arrives: the server hands it the tenants it has not served before, after the checks pass them, so the host can ready what a tenant needs, such as its first key, before the first request arrives. A tenant that is not readied is left out of that reading and handed over again at the next one, and a tenant declared in the options that is not readied at startup stops the server from starting.

A tenant that disappears from the store stops being served, but what it kept outside the server's memory has to be let go too. The server releases a tenant at the first reading at least one refresh period after a reading first found it gone, so between one and two refresh periods after the removal, and hands every tenant that reading releases to each `ITenantClosing` in one call, so a closing that cleans a shared store does it in one pass.

Each tenant is passed with the definition last served for that creation, and the call gets a cancellation token that fires one refresh period after the call begins. The reading waits for the call, so a closing should honor the token.

The call returns the tenants it could not close, each with the reason. Those, and every tenant of a call that throws or runs out of time, are logged with the tenant's id and generation and are not handed over again, while the other tenants and closings still run. Closing covers the tenants an instance has read from the store, served or not; a tenant never served arrives with the definition it was first listed in. For a tenant removed while no instance ran, cleanup is up to the host.

When a tenant is removed:

- The server deletes the keys it minted for that creation, and with the Shared Signals package, the tenant's streams and the events queued on them.
- Codes, sessions, and the records of revoked and already-used tokens expire on their own lifetimes; a tenant created again under a new generation cannot reach them.
- The host deletes the rest: the client registrations it stores, its users and consents, any settings it keeps per tenant, the keys a custodian holds for the tenant, and everything of a tenant removed while no instance ran.

:::caution[Removing a tenant is not an instant cut-off]
Each instance stops serving a removed tenant at its next successful reading, and an instance that cannot read the store keeps serving it until a reading succeeds. The access tokens the tenant issued stay valid at resource servers until they expire. To cut a tenant off at once, block its issuer at the proxy, and keep token lifetimes short.
:::

## Keys and tokens through changes {#keys-and-tokens-through-changes}

Where a tenant's keys rotate depends on the choice the server makes for all of them.

Keys the server mints live in a ring of their own for each tenant and rotate on the ring's schedule; the first key is minted before the tenant is first served (see [when a tenant arrives](#when-a-tenant-arrives-and-leaves)), and a tenant that is removed is no longer rotated. The schedule belongs to the server, not to the custodian, which only protects the keys:

- A new key is minted every 30 days by default, set by `RotateEvery` on the `MintedKeys` policy the host passes when it registers the key ring.
- The server publishes a new key an hour before signing with it, so that consumers with a cached key set have seen it. The hour is `OidcOptions.KeyRolloverPropagation`.
- A retired key is kept for one more rotation period, or for `KeepRetiredFor` on the same policy when it is set.

A retired key must outlast every token it signed, refresh tokens included: when it leaves the key set, the unexpired tokens it signed stop verifying.

Keys declared in `SigningKeys` change when the host changes the tenant's definition, and the new definition reaches the server through the manager or at the next reading of the store. The server publishes exactly what the definition holds at that moment and signs with the first key of a permitted algorithm, with no retention period of its own. Rotate in steps, each one a change of the definition, because each instance serves a change only from its next reading, and relying parties cache the key set:

- Add the new key after the current one.
- Wait longer than the refresh period plus the time relying parties cache the key set, then move the new key first.
- Remove the old key only after the last token it signed has expired; removing it earlier invalidates those tokens at once.

A rotated client secret meets the same window: until every instance has read the change, some instances still accept only the old secret.

Keys held by a custodian never leave it, and they rotate there, on the custodian's own schedule: the server publishes every version of the named key under that version's own key id, which lets a rotation overlap and sends each private operation back to the exact version that signed.

When a tenant is released, the server lets go of its key ring in memory and deletes the keys it minted for that creation from the key store, so a deployment that creates and removes tenants does not pile up key material nobody reads. A tenant created again under the same id mints into the partition of its new generation and inherits nothing. Keys from the settings or from a custodian are not the server's, and it deletes none of them.

Changing a tenant keeps its generation, so what it issued before the change is still its own, and a token stays verifiable for as long as the key that signed it is published. Removing a tenant ends it: requests to its addresses are answered as for an unknown tenant, and what the server held for it in memory is let go when it is released, one to two refresh periods later.

## Work outside a request {#work-outside-a-request}

Inside a request, the tenant accessor (`ITenantAccessor`) answers with the tenant the request was resolved to. Work that runs outside a request has no such tenant, so the services a host calls itself take the tenant as their first argument: `ITokenRevoker` revokes tokens, `IAuthSessionTerminator` ends a session, `IUserCodeVerificationService` approves or denies a device code, and `IAuthenticationCompletionHandler` completes or denies a back-channel authentication. Outside a request, pass them the definition `ITenantCatalog` returns. In this sample an administration job revokes a subject's tokens at Acme, with the catalog and the revoker taken from the container:

```csharp
using Abblix.Oidc.Server.Features.MultiTenancy;          // ITenantCatalog
using Abblix.Oidc.Server.Features.Tokens.Revocation;     // ITokenRevoker

var tenant = await catalog.FindByIdAsync("acme", cancellationToken)
    ?? throw new InvalidOperationException("No tenant is registered under the id acme.");

await tokenRevoker.RevokeSubjectAsync(tenant, subject, cancellationToken: cancellationToken);
```

A service that takes no tenant of its own reads it from the accessor, and outside a request it runs inside `TenantScope.Enter`, which makes a tenant the current one for a block of work. The Shared Signals event dispatcher is one, and it is also taken from the container inside the scope, because the dispatcher the container hands out belongs to the tenant served at that moment ([Shared Signals per tenant](#shared-signals-per-tenant)). An event dispatched from a message consumer, with `services` the application's service provider:

```csharp
using Abblix.Oidc.Server.Features.MultiTenancy;          // TenantScope
using Abblix.SharedSignals.Transmitter;                  // EventDispatcher

using (TenantScope.Enter(tenant))
{
    var dispatcher = services.GetRequiredService<EventDispatcher>();
    await dispatcher.DispatchAsync(descriptor, cancellationToken);
}
```

A service of your own that reads `ITenantAccessor` runs inside a scope the same way.

The scope flows across awaits and ends when disposed. Entered inside a request, it takes precedence over the request's tenant. Await the work you start inside a scope before the scope ends: work started inside it and still running afterwards keeps running as its tenant. An operation that needs a tenant's data and has none, outside a request and outside a scope, fails with an `InvalidOperationException` rather than falling back to a default tenant.

## Seeing tenants in telemetry and logs {#telemetry-and-logs}

The server's endpoint spans and its request and token metrics carry the tenant's id under the `oidc.tenant` tag (`TelemetryTags.Tenant`). Log records do not. To find a tenant's requests in the log, add a logging scope right after `UseMultiTenancy`:

```csharp
app.UseMultiTenancy();
app.Use(async (context, next) =>
{
    var tenant = context.RequestServices.GetRequiredService<ITenantAccessor>().Current?.Tenant;
    var logger = context.RequestServices.GetRequiredService<ILoggerFactory>().CreateLogger("Tenant");
    using (logger.BeginScope("Tenant {TenantId}", tenant?.Id))
        await next(context);
});
```

A structured provider records `TenantId` as a property of every record written during the request. A provider writes the scope only when it is set to include scopes, as `IncludeScopes` does for the console logger.

## What stays yours {#what-stays-yours}

The server partitions what it owns. The users and the consents are yours, and the sign-in session belongs to its tenant, as the caution in [Serve two tenants](#turn-it-on) describes. This section covers the client registrations a host keeps itself and the registries a host may bring.

A host that keeps the clients added by [dynamic client registration](https://www.abblix.com/en/docs/glossary-overview#dynamic-client-registration) in its own database implements `ITenantClientRegistrationStore`, registered as a singleton, whose methods all take the tenant of the request. Its contract:

- A registration is kept and found under the tenant's id and generation together, so a registration made at one tenant is never found at another, and a recreated tenant finds none of the earlier one's.
- Client ids are compared without regard to case, as the server compares them.
- The clients a tenant's definition lists stay in the definition and win over a registration under the same id. The server removes, through `TryRemoveAsync`, a registration it meets under such an id.
- The registrations of a tenant the host removes are the host's to delete: the server only stops asking for them.

A host can also bring a registry of its own: the client registry (`IClientInfoProvider` and `IClientInfoManager`), `IScopeManager`, `IResourceManager`, `ISubjectTypeConverter`, and the keys provider. Each of them is handed the tenant of the request as its first parameter, so it answers from that tenant's set; one that ignored it would let a client registered at one tenant authenticate at every other. A client registry of your own replaces the server's whole answer, including the clients a definition lists, so to keep only the dynamically registered clients in your database, the registration store above is the smaller change.

## Moving a server from version 2 {#moving-an-existing-server}

A version 2 server serves one issuer from the settings on `OidcOptions`. In 3.0 that issuer is one tenant, declared with the same address. The upgrade does not carry over what the server stored, so signed-in users sign in again and refresh tokens issued by version 2 are refused. [Upgrading to 3.0](https://www.abblix.com/en/docs/upgrading-to-3.0) lists everything the upgrade ends, and how to keep the signing keys the server minted.

## Licensing {#license}

The free tier serves one issuer, and every other protocol and profile is available on every tier. Each tenant is an issuer, so the license has to allow at least as many issuers as the tenants you serve; the [pricing page](https://www.abblix.com/en/oidc-server/pricing) says how to get such a license. A license is supplied in `OidcOptions.LicenseJwt`, and the [configuration guide](https://www.abblix.com/en/docs/configuration-and-setup#apply-the-license) covers it. What a running server does with it:

- Tenants declared in the options are checked at startup, once the license is loaded: the server does not start when one of them is at an issuer the license does not name, or when they outnumber the issuers it allows. Tenants from a store of your own are not checked at startup; what the license does not cover among them is refused at request time, with 500.
- The license counts tenants in the order of their first requests. A tenant that asks once the count is full is refused, and the tenants already counted are served; the one refused is usually the newest, but not always. A license that narrows or expires beneath the tenants already counted refuses them all.
- A license that names its issuers serves only those. The manager refuses to create a tenant at another issuer, or to move one there, with `IssuerNotLicensed`.
- A tenant counts once whatever address it moves to. A tenant the store no longer holds, together with its clients when the license limits clients, keeps its place until the server releases it, one to two refresh periods after the store lost it.
- The manager counts only the tenants the store holds. At a full license, wait two refresh periods between removing one tenant and creating another; created sooner, the new tenant finds the count full, and its requests are refused until the removed one is released.

## Shared Signals per tenant {#shared-signals-per-tenant}

A [Shared Signals](https://www.abblix.com/en/docs/shared-signals-framework) transmitter that answered for every tenant alike would let one tenant's receivers see another's streams, and would sign events in the name of the wrong issuer. The `Abblix.OIDC.Server.SharedSignals` package runs the transmitter per tenant instead. Register the transmitter and its stores first, then chain `AddSharedSignals()` onto the builder that `AddOidcServices` returns. A tenant declares its own streams with `AddConfiguredStreams(tenantId, streams)` on the same builder, after `AddSharedSignals()`.

Each tenant then transmits as itself. Its streams, the events queued on them, and the claims on their delivery are kept apart from the others' in whatever stores the transmitter uses, so two tenants may declare a stream under the same receiver and stream ids. The issuer, key set address, and endpoint addresses its receivers see are its own, every security event token is signed with the keys of the tenant it is transmitted for, and push delivery runs for each tenant the server serves. Signing and addresses follow the definition the server serves for the tenant, so once an instance serves a rotated key, the events it sends are signed with that key. When a running instance releases a tenant, its streams and the events queued on them are deleted through `ITenantClosing`. With the Redis stream store, each tenant's streams are kept on a hash of the tenant's own, so each dispatched event and each tenant's delivery pass read that tenant's streams alone. A store you wrap around it implements `IPartitionedStreamStore` (namespace `Abblix.SharedSignals.Transmitter`) and returns the inner store's partition wrapped, or the server does not start.

Each tenant has a transmitter of its own, and what your code takes from the container, such as the event dispatcher, the stream store, the event outbox, or the stream management service, is the part of the transmitter of the tenant served at that moment. Take it in the request of the tenant the event belongs to, or inside `TenantScope.Enter` for that tenant, as [work outside a request](#work-outside-a-request) shows. Until its tenant is released, a part kept across requests throws an `InvalidOperationException` once it is used where its tenant is not served, instead of putting one tenant's events on another's streams or signing them with another's keys.

A tenant catalog of your own hands every request the tenant's definition in force, and your code enters every tenant scope with that definition too: the transmitter signs and answers as the definition it was last served, so a stale one brings back the keys and the issuer it names. Each push delivery pass takes every tenant's definition from your catalog at that moment, so pushed events follow a move or a key change without waiting for a request, and a tenant your catalog no longer serves is pushed nothing. When such a catalog moves a tenant to another issuer, the streams the tenant declares are answered with the new issuer and a poll address under it, while their stored records keep the issuer and the poll address they were first written with until the server restarts.

The [package README](https://github.com/Abblix/Oidc.Server/blob/master/src/Abblix.Oidc.Server.SharedSignals/README.md) has the complete registration. The server does not start when the stream store, the event outbox, the transmitter's identity, or its delivery pass is replaced after `AddSharedSignals()`, when the transmitter's own issuer option names a path, when its key set address is relative or on another host, or when streams are declared for the whole server rather than for a tenant, since such streams belong to no tenant's receivers. A server whose tenants come from its settings alone also stops, once it listens, when streams are declared for a tenant id the settings do not declare. Tenants whose issuers have paths are served as usual.

## Working examples {#working-examples}

The Abblix.Oidc.Server repository holds a suite that runs two tenants against a real host: [`Abblix.Oidc.Server.MultiTenancy.E2E.Tests`](https://github.com/Abblix/Oidc.Server/tree/master/tests/Abblix.Oidc.Server.MultiTenancy.E2E.Tests). `TenantManagementTests` creates, changes, and removes tenants through the manager while the server runs; `TenantIsolationTests` checks, among other things, that a client, a scope, a pushed request, or a device code of one tenant is unknown to the other, that pairwise subjects differ from tenant to tenant, and that a token is verified by its own tenant's keys alone; the Shared Signals suites cover the transmitter per tenant. The host in this article is the composition those tests build, with the test scaffolding taken out.
