We replaced our registry mirror scanner with a K8s admission webhook

Last week we built a registry mirror scanner that copies images to a local zot registry and patches deployments every 5 minutes. It worked, but it was annoying.

Deployments rolled out twice. First with the original image, then again after the scanner patched it. Flux kept reverting our patches because the live state didn't match git. And if the scanner was down, new deployments just pulled from Docker Hub anyway.

So we rebuilt the whole thing as a K8s admission webhook.

How it works now

We intercept deployments at the K8s API before they exist. The API server sends every deployment CREATE/UPDATE through our webhook, we rewrite nginx:latest to registry-mirror.tinysystems.io/library/nginx:latest, and hand it back. K8s creates the deployment with the local image. One rollout. Git untouched.

We also turned zot into a pull-through cache. When a pod tries to pull from the local registry and the image isn't there yet, zot goes and gets it from Docker Hub (or ghcr, quay, gcr) on the fly. No more copying images around.

The flow

Six nodes. Click Start.

Ticker -> Cert Generate -> Webhook Register -> HTTPS Server
                                               HTTPS Server request -> JSON Decode -> JS (rewrite) -> HTTPS Server response

The Ticker fires with your config. Cert Generate makes a self-signed ECDSA cert with the right SANs for the in-cluster service. Webhook Register creates the MutatingWebhookConfiguration pointing at the HTTPS server, with the CA bundle from that cert. Then the HTTPS server starts and blocks, waiting.

When someone creates or updates a deployment, K8s sends an AdmissionReview to our server. The JS checks each container image, skips anything already pointing at the local registry, rewrites the rest, and returns a JSON patch.

We run with failurePolicy: Ignore. If the flow is down, deployments go through normally. Nothing breaks.

Zot with pull-through cache

Previously we had to copy images to zot before rewriting. That step is gone. Zot's sync extension with onDemand: true fetches on first pull.

Here's the values.yaml:

persistence: true
pvc:
  storage: 20Gi
mountConfig: true
configFiles:
  config.json: |
    {
      "storage": {
        "rootDirectory": "/var/lib/registry",
        "dedupe": false,
        "gc": true,
        "gcDelay": "1h",
        "gcInterval": "6h"
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "compat": ["docker2s2"]
      },
      "log": { "level": "info" },
      "extensions": {
        "search": { "enable": true },
        "scrub": { "enable": true, "interval": "24h" },
        "sync": {
          "enable": true,
          "registries": [
            {
              "urls": ["https://registry-1.docker.io"],
              "onDemand": true,
              "tlsVerify": true
            },
            {
              "urls": ["https://ghcr.io"],
              "onDemand": true,
              "tlsVerify": true
            },
            {
              "urls": ["https://quay.io"],
              "onDemand": true,
              "tlsVerify": true
            },
            {
              "urls": ["https://gcr.io"],
              "onDemand": true,
              "tlsVerify": true
            }
          ]
        }
      }
    }
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: registry-mirror.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: registry-mirror-tls
      hosts:
        - registry-mirror.example.com

The new part compared to the previous post is the sync section. Four upstream registries, onDemand: true. Zot sees a pull for something it doesn't have, fetches it, caches it. First pull is slower, everything after that is local.

You still need "compat": ["docker2s2"] in http. Without it zot rejects Docker V2 Schema 2 manifests with a confusing MANIFEST_INVALID error. We lost time on that one.

What we had to build

Two things didn't exist in Tiny Systems before this.

cert_generate is a new component in the crypto-module. Generates a self-signed ECDSA P-256 cert with whatever CN and SANs you give it. Outputs PEM cert, PEM key, and a base64 CA bundle you can feed straight into a webhook config. We were running openssl manually before and pasting certs into forms. Not fun.

webhook_register is in the kubernetes-module. Creates or updates a MutatingWebhookConfiguration. You give it the service name, port, CA bundle, failure policy, and what resources to intercept. We also added TLS support to the http_server component so it can serve HTTPS with a cert/key pair from the flow.

The admission webhook pattern is now something you can wire up visually in Tiny Systems without touching kubectl.

Scanner vs webhook

The scanner still makes sense if you have existing deployments to migrate. For everything going forward, the webhook is simpler.

Try it

The Image Policy Webhook is on the Tiny Systems marketplace. You need:

• Zot with sync enabled (values above)
• kubernetes-module v0.5.2+
• http-module v0.2.2+
• crypto-module v0.1.0+

Fill in your local registry, pick a namespace or leave empty for all, set which registries to rewrite. Start the ticker.