Query-parameter based stickiness in an nginx ingress

There is one problem which caused me a lot of headache lately. But let me first introduce you to the context and the source of all evil. The requirement was to find a way to create stickiness based on a substring of a query-parameter in an Nginx Ingress controller. When set up, each call with the same query-parameter substring should be forwarded/directed to the same pod. There are several reasons for such a requirement but it is mostly the performance which should be improved by this feature.

My approach is quite trivial but a result of reading tons of documentation. The idea was to use the following example set-up explained in more detail below.

Deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app-deployment
  labels:
    app:
spec:
  replicas: 8
  selector:
    matchLabels:
      app: hello-app
  template:
    metadata:
      labels:
        app: hello-app
    spec:
      containers:
      - name: hello-app
        image: gcr.io/google-samples/hello-app:1.0

The deployment is basically a composition of eight pods with the hello-app container from the google-samples. If this deployment is exposed on port 8080 and the hosts file contains an entry which points hello-world.info to the exposed IP, a simple curl hello-world.info:31281 will return the following response:

Hello, world!
Version: 1.0.0
Hostname: hello-app-deployment-6bb444b4f4-6zxh7

And here we want the same Hostname for calls with the same query-parameter substring while other calls are forwarded to other pods as well.

Ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    nginx.ingress.kubernetes.io/http-snippet: |
      map $args_apikey $shortApikey {
        default                "";
        "^(?<short>.{32}).*$"  $short;
      }
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/upstream-hash-by: "$shortApikey"
spec:
  rules:
    - host: hello-world.info
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: hello-app-deployment
                port:
                  number: 8080

The Ingress controller is built in a very clean way. The idea is that every path (.*) will redirect to itself by using the nginx.ingress.kubernetes.io/rewrite-target: /$1 annotation. And each call should have a stickiness based on the hashed value of $shortApikey by using nginx.ingress.kubernetes.io/upstream-hash-by: "$shortApikey". The extraction of the substring should take part in the map-directive inside the http-snippet annotation. The other settings in this Ingress config should be trivial. An example call which should be always routed to the same pod is: curl hello-world.info/example?apikey=1234567890abcdefghijhlmnopqrstuvwxyz. Here, the apikey parameter should be cut down to 32 characters (1234567890abcdefghijhlmnopqrstuv) and stored inside the $shortApikey.

The problem with this setup is the fact that http-snippets defined as annotations are not written into the http section of the controllers Nginx config. And as Nginx allows map-directives in the http context only, there is no other way around. To double check this behavior, I created a ConfigMap in the same namespace and used the http-snippet there. Same problem. Long story short, I am able to get snippets into the Nginx config location and server section but not into the http section. Searching online for a solution did not help.

Reduced Approach:

With this setback in mind, the only working way was to use upstream-hash-by with the complete query-parameter apikey. This will use the complete key and the application needs to be adapted to always send the full key but in the end, it is working that way. It still does not feel satisfying as the documentation shows the possibility of using map-directives as part of a http-snippet annotation inside a config-map.

Finally, a working Ingress controller which offers stickiness based on the the complete value of a query-parameter is displayed below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/upstream-hash-by: "$args_apikey"
spec:
  rules:
    - host: hello-world.info
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: hello-app-deployment
                port:
                  number: 8080

It somehow feels that I will come back to the problem at some point in the future but for now, I will focus on other things as there is one more problem to solve when using a config-map. If the http-snippet works and I am using the load-balance config-map key to define the upstream-hash-by, I will have a conflict with the variables created in the different entries of the config-map. Even though variable in Nginx are always global…