Identity Server 4 On Kubernetes Nginx Ingress

Have you ever tried deploying Identity Server 41 on a k8s (Kubernetes2) setup with Nginx3 ingress?

If you tried, I’m sure you’ve encountered some problems, as the current Nginx ingress is not properly configured for ASP.Net projects or does not contain better optimization for Identity Server 4. The symptoms are frustrating: tokens that arrive truncated, discovery documents that hang, and redirects that bounce between HTTP and HTTPS like a confused ping pong ball.

The first step towards getting somewhere is to decide you’re not going to stay where you are.

~ J.P. Morgan.

Come on, join me as we dive into the configurations!

✅ Prerequisites

First of all, you must have a Kubernetes on your machine. Second, must have an existing test bed project for Identity Server 4.

If you don’t have Kubernetes, perhaps you could try installing MicroK8s. The MicroK8s works on Windows and MacOS, and its single command install makes it the fastest way to get a cluster at home.

🛠️ So where do we start?

First, we modify the ingress ConfigMap configuration, and add the following lines:

proxy-buffer-size: "128k"  
proxy-buffers: "4 256k"  
proxy-busy-buffers-size: "256k"  
client-header-buffer-size: "64k"  
http2-max-field-size: "16k"  
http2-max-header-size: "128k"  
large-client-header-buffers: "8 64k"

This specific modification allows Identity Server 4 to send and receive large header data which is needed to store and sort out JWT (JSON Web Token) identifiers. By default, Nginx buffers are sized for ordinary web pages, and the OpenID Connect dance, with its fat ID tokens and long protocol URLs, blows right past those defaults. Bigger buffers fix that class of problems entirely.

You can check this sample setup on my test ingress config map YAML (Yet Another Markup Language):

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-nginx-ingress
  namespace: default
  selfLink: /api/v1/namespaces/default/configmaps/nginx-ingress-nginx-ingress
  uid: 9fe8c06b-4f7c-4032-a938-505c308ed332
  resourceVersion: '10291469'
  creationTimestamp: '2020-09-18T12:46:50Z'
  labels:
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: default
data:
  client-header-buffer-size: 64k
  http2-max-field-size: 16k
  http2-max-header-size: 128k
  keepalive-timeout: '65'
  large-client-header-buffers: 8 64k
  proxy-buffer-size: 128k
  proxy-buffers: 4 256k
  proxy-busy-buffers-size: 256k
  proxy-http-version: '1.1'
  proxy-read-timeout: '150'
  sendfile: 'on'
  use-http2: 'false'

Next thing we do is adjust our code to forward headers from and to the ingress. The other method calls are also recommended by the docs from Microsoft, you can check the setup here:

public void ConfigureServices(IServiceCollection services)  
{
    // ... code omitted ...
    // Needed for load balancer to forward headers
    services.Configure<ForwardedHeadersOptions>(options =>
    {
        options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
        options.RequireHeaderSymmetry = false;
        options.KnownNetworks.Clear();
        options.KnownProxies.Clear();
});

The docs specified the known networks and proxies are needed if you are hosting C# apps in a non-Windows hosting environment. Clearing them tells ASP.Net to trust whatever proxy sits in front, which in this setup is the Nginx ingress. Without this, the app sees the ingress IP instead of the real client, and HTTPS detection goes haywire.

After adding a forward headers configuration onto our ConfigureService method, we also need to add the forward headers middleware on the Configure method, which can also be found in Startup.cs file:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    // ... code omitted ...
    app.UseForwardedHeaders();
    // ... code omitted ...
}

Then after that, restart the Nginx ingress and also your app to test whether everything is working fine. The next change is optional if you are using TLS.

If your ingress setup is TLS4 terminated, you also need to add this on your Configure method:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    // ... code omitted ...
    app.Use(async (ctx, next) =>
    {
       ctx.Request.Scheme = "https";
       await next();
    });
    // ... code omitted ...
}

This specific custom middleware converts all incoming calls to the secured HTTP scheme. The TLS ingress specifically does is redirect the calls from your RS (Resource Server) to AS (Authorization Server) which is Identity Server 4, but TLS needs a consistent HTTP secured scheme. If you look into your openid-configuration it will return http:// only endpoints, and that is the problem, and that’s why we are modifying it internally using a custom middleware. Once the scheme is forced to https, every discovery doc, JWKS endpoint, and redirect in the flow points at the secure URL, and the whole protocol works as designed.

After all is done, restart the service and test every knick and knack.
That’s all guys!

🧠 Final Thoughts

It’s not just a simple clone image and deploy setup in k8s, especially if you’re trying to deploy a C# app. Sometimes you need to optimize some config in order for it to run smoothly and work well. The three adjustments here, Nginx buffers, forwarded headers, and the HTTPS scheme override, cover the vast majority of Identity Server pain on Kubernetes, and each one is a ten minute fix that saves you days of head scratching.

Check the recommended deployment guide in Microsoft docs, and keep this checklist handy for the next time your tokens come back truncated, because it will happen again.

Let me know in the comments if you have questions or queries, you can also DM me directly.

Follow me for similar article, tips, and tricks ❤.

1. IdentityServer is an OpenID Connect provider: it implements the OpenID Connect and OAuth 2.0 protocols. ↩︎
2. Kubernetes is an open-source container orchestration system for automating computer application deployment, scaling, and management. ↩︎
3. Nginx (pronounced “engine X”, /ˌɛndʒɪnˈɛks/ EN-jin-EKS), stylized as NGINX, nginx or NginX, is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. ↩︎
4. Transport Layer Security (TLS), and its now-deprecated predecessor, Secure Sockets Layer (SSL), are cryptographic protocols designed to provide communications security over a computer network. Several versions of the protocols are widely used in applications such as web browsing, email, instant messaging, and voice over IP (VoIP). Websites can use TLS to secure all communications between their servers and web browsers. ↩︎