We built a self-healing registry mirror (because Docker Hub rate limits are no fun)

If you've ever stared at ImagePullBackOff in your cluster at 2 PM on a Tuesday, you know. Docker Hub rate limits hit, your pods can't pull, and a perfectly fine deployment is stuck.

We set up a local registry mirror that copies images from remote registries and patches deployments to use the local copies. Rate limits gone. Surprise outages gone.

Zot on GKE

We went with zot — lightweight, OCI-native, runs as a single StatefulSet. Install with Helm:

helm repo add zot https://zotregistry.dev/helm-charts

Our values.yaml:

persistence: true
pvc:
  storage: 20Gi
mountConfig: true
configFiles:
  config.json: |
    {
      "storage": {
        "rootDirectory": "/var/lib/registry",
        "dedupe": false,
        "gc": true,
        "gcDelay": "1h",
        "gcInterval": "6h"
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "compat": ["docker2s2"]
      },
      "log": { "level": "info" },
      "extensions": {
        "search": { "enable": true },
        "scrub": { "enable": true, "interval": "24h" }
      }
    }
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: registry-mirror.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: registry-mirror-tls
      hosts:
        - registry-mirror.example.com

helm install zot zot/zot -n zot --create-namespace -f values.yaml

Two things to watch for.

The docker2s2 gotcha

Zot is OCI-first and rejects Docker V2 Schema 2 manifests by default. You get MANIFEST_INVALID or HTTP 415 when pushing Docker Hub images.

"compat": ["docker2s2"] in the http section fixes it. Took us a few rounds to find — it goes under http, not storage. The error message gives you nothing to work with.

Keep it internal

The mirror sits inside the cluster, no reason to expose it. The whitelist-source-range annotation locks access to your pod CIDR. Adjust the range for your cluster. Pods pull freely, nobody else gets in.

The automation

A registry without automation is just storage. We built a Tiny Systems flow that runs every 5 minutes:

• Lists deployments (filterable by namespace and labels)
• Skips anything already using the local registry, or scaled to zero
• Reads each deployment's own imagePullSecrets for source auth
• Copies the image to the local mirror
• Patches the deployment to use the local copy

Nine nodes, no code to deploy, no CronJob YAML.

It handles the stuff you'd forget in a script: deployments without pull secrets skip the secret read. Multi-container pods get each container mirrored separately. Failed copies don't block the rest.

Ticker (5min) -> Deployment List -> JS (plan) -> Split -> Router -> HAS_SECRET -> Secret Get -> Registry Copy -> Update
                                                                 -> NO_SECRET  -> Registry Copy -> Update

The Router splits on whether the deployment has imagePullSecrets. Private images go through Secret Get first. Public images go straight to copy.

What happened

After starting the ticker, every deployment in our namespace got mirrored within a couple of minutes. Docker Hub, ghcr.io, quay.io — all local.

Next time Docker Hub has a bad day, our cluster won't notice.

Update: we've since replaced this scanner with an admission webhook that rewrites images at the K8s API level — no double rollouts, no GitOps drift.

Try it

The Image Mirror solution is on the marketplace. Set your local registry in the Ticker settings, click Start.

You need these modules installed:

• common-module — ticker, router, array split, debug
• kubernetes-module — deployment list/update, secret get
• js-module — image planning logic
• distribution-module — registry copy