Pushing images to Docker registry with Pull-Through Cache enabled

Sat, Apr 5, 2025

Rational

Configuring Docker registry as a pull-through cache isn’t something new. The Docker Registry Distribution, supports proxying docker.io for quite some time now. The only caveat being, once you configure it for pull-through caching, you cannot use it to push images any more. Almost all other 3rd party docker registries available(Harbor, Nexus, Quay, Artifactory) support this feature, allowing you to pull-through and also push. But they are a lot more complex to configure run and operate. Today we will configure docker registry for pull through caching but we will be able to push too!

Let’s start

Some context first. On my home lab I am running a Docker Registry for storing all my docker images. I am using the official Docker Registry Distribution that very recently released their 3 version. Everything started a few days ago, after troubleshooting some self hosted applicationon on my k3s cluster, when I got rate-limited by Docker.io. I thought maybe its time I do it more elegantly by centraly caching those images when I pull them.

I searched the docker registry documentation and figured it out very very quickly. All I needed to do was, add this small snippet in the configuration:

delete:
    enabled: true
proxy:
    remoteurl: https://registry-1.docker.io
    username:
    password:

The delete config is required for the scheduller to be able to delete manifests periodicaly, at least for the proxy feature. The remoteurl is the docker.io registry url along with your docker.io account’s username and password. Since I am runnig the registry on k8s, I moved the username/password config to secrets like so:

kind: Secret
type: Opaque
apiVersion: v1
metadata:
    name: registry-credentials
    namespace: registry
stringData:
    access_key: my minio access key
    secret_key: my minio secret key
    proxy_username: [email protected]
    proxy_password: myverysecretpassword

At this point I have to point out that I also use minio/s3 as the registry storage backend. This may sound irrelevant now but it will be make more sense later on...

I added the secrets in the docker registry’s deployment.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: registry
  name: registry
spec:
  selector: 
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
      name: registry
    spec:
      containers:
        - image: "registry:3"
          name: registry
          ports:
            - containerPort: 5000
          env:
            - name: OTEL_TRACES_EXPORTER
              value: none
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
            - name: REGISTRY_STORAGE_S3_ACCESSKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: access_key
            - name: REGISTRY_STORAGE_S3_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: secret_key
            - name: REGISTRY_PROXY_USERNAME # Here
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: proxy_username
            - name: REGISTRY_PROXY_PASSWORD # and here...
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: proxy_password
          volumeMounts:
            - name: config-volume
              mountPath: /etc/distribution/
      volumes:
        - name: config-volume
          configMap:
            name: registry-config
            items:
            - key: config.yml
              path: config.yml

The Official registry distribution only supports proxying docker.io but that’s ok. What I did now is, I configured the registry to always proxy/cache requests for hub.docker.io. Now I needed to configure k3s to transparently use this registry as a mirror for hub.docker.io by configuring the following file: /etc/rancher/k3s/registries.yaml

mirrors:
  docker.io:
    endpoint:
      - "https://registry.mydomain.com"
  registry.mydomain.com:
    endpoint:
      - "https://registry.mydomain.com"

This tells k3s to first look for any docker.io image in my registry at registry.mydomain.com first and it they are not there, to pull them and cache them there. So next time k3s tries to pull the same image from docker.io it will search it and find it in my registry instead, saving me time and bandwidth.

After that all I did was to restart k3s and the magic just happened. The registry started filling with the images I was pulling from docker.io.

The bummer

At this point I was very happy because I had not yet realized that, by using this configuration, I couldn’t push to my registry any more…

time="2025-04-12T09:41:38.107096663Z" level=error msg="response completed with error" err.code=unsupported err.message="The operation is unsupported." go.version=go1.23.7 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=registry.mydomain.com http.request.id=5dd230ca-a780-4988-8e24-6ef64c992953 http.request.method=PUT http.request.remoteaddr=172.30.1.1 http.request.uri=/v2/moby/buildkit/manifests/buildx-stable-1 http.request.useragent="docker/28.0.4 go/go1.23.7 git-commit/6430e49 kernel/6.13.9-200.fc41.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.4 \\(linux\\))" http.response.contenttype=application/json http.response.duration=1.423523ms http.response.status=405 http.response.written=78 instance.id=fe9fce5c-4135-4c57-94fe-d3a46ba06555 service=registry vars.name=moby/buildkit vars.reference=buildx-stable-1 version=3.0.0

Why? Because as always I didn’t read the documentation:

The proxy structure allows a registry to be configured as a pull-through cache to an upstream registry such as Docker Hub. See mirror for more information. Pushing to a registry configured as a pull-through cache is unsupported.

from: https://distribution.github.io/distribution/about/configuration/#proxy

What a bummer…

Lets start. Again…

The first thing I did ofcourse was to revert everything so I could push to my registry again. I was very disapointed with myself and with the docker.io because deep down I thought there should be some technical complexity that enforces this limitation. After I stoppped crying, I started to think about any possible solutions.I dont want to go for another registry distribution, like harbor or Nexus because I dont wanna… I like my registry as I like my coffee… black. No that doesnt make any sense. Forget it. I like this registry because its stateless and really requires nothing more than storage backend to run.

My first thought was to check the docker registry api. A quick documentation read, revealed that pushing to registry is either a PUT, PATCH or POST method. Maybe I can create 2 registry deployments and split pulls/pushes requests between them. One deployment with proxy enabled will get the pull requests for caching and the other one, without proxy enabled will get the push requests. Both of them with the same storage backend configured, minio!

How can I do that with Ingress? I use Traefik Ingress controller. Method routing is not supported by Ingress. The first possible solution was Traefik’s ingressRoute CRD, But it did seem a bit complex and not very portal if I later on ewanted to move away from Traefik.

Then my very good friend Spyros who I discused this with, suggested to have a look at Kubernetes Gateway API which, maybe that supports method routing and for my surprise Yes it does…

kubernetes Gateway API to the rescue

So what’s Gateway API.

Gateway API is an official Kubernetes project focused on L4 and L7 routing in Kubernetes. This project represents the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs. From the outset, it has been designed to be generic, expressive, and role-oriented.

from: https://gateway-api.sigs.k8s.io/

Very quickly, gateway api implemenation is spliting the Ingress k8s resource in to some new k8s resources. The bare minimum resources required for translating an Ingress resource are: A gateway resource which is the actual entrypoint to the cluster. You can have one gateway for the whole cluster or one per application. A httpRoute resource that implements some layer 7 rules for routing traffic from the gateway to the actual application.

In my case decided to update my cluster’s Traefik helm installation and reconfigure it with gateway api enabled. This created a new gateway resource that will listen for httpRoute resources across all namespaces.

gateway:
  enabled: true
  name: ""
  namespace: ""
  annotations: {}
  infrastructure: {}
  listeners:
    web:
      port: 80
      hostname: ""
      protocol: HTTP
      namespacePolicy:  "All" # @schema type:[string, null]
    websecure:
      port: 443
      protocol: HTTPS
      namespacePolicy: "All"
      certificateRefs: 
        - name: my-wildcard-tls-certificate-secret

Back to the registry. All I needed to do now was to create 2 registry deployments one with proxy feature enabled and one without:

One configmap with 2 configurations:

apiVersion: v1
kind: ConfigMap
metadata:
  name: registry-config
data:
  config.yml: |-
    version: 0.1
    delete:
      enabled: true
    proxy:
      remoteurl: https://registry-1.docker.io
    log:
      fields:
        service: registry
      level: info
    storage:
        # cache:
        #     blobdescriptor: inmemory
        s3:
            accesskey: awsaccesskey
            secretkey: awssecretkey
            regionendpoint: https://my.minio.url
            bucket: docker-registry
            region: us-west-1
            chunksize: 5242880
            rootdirectory: /
            forcepathstyle: true
    http:
        addr: :5000
        headers:
            X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3
  config-push.yml: |-
    version: 0.1
    log:
      fields:
        service: registry
      level: info
    storage:
        s3:
            accesskey: awsaccesskey
            secretkey: awssecretkey
            regionendpoint: https://my.minio.url
            bucket: docker-registry
            region: us-west-1
            chunksize: 5242880
            rootdirectory: /
            forcepathstyle: true
    http:
        addr: :5000
        headers:
            X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3

Two Services:

kind: Service
apiVersion: v1
metadata:
  name: registry
spec: 
  selector: 
    app: registry
  type: ClusterIP
  ports:
    - name: api-port
      port: 5000
      protocol: TCP
      targetPort: 5000
---
kind: Service
apiVersion: v1
metadata:
  name: registry-push
spec: 
  selector: 
    app: registry-push
  type: ClusterIP
  ports:
    - name: api-port
      port: 5000
      protocol: TCP
      targetPort: 5000

Two Deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: registry
  name: registry
spec:
  selector: 
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
      name: registry
    spec:
      containers:
        - image: "registry:3"
          name: registry
          ports:
            - containerPort: 5000
          env:
            - name: OTEL_TRACES_EXPORTER
              value: none
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
            - name: REGISTRY_STORAGE_S3_ACCESSKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: access_key
            - name: REGISTRY_STORAGE_S3_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: secret_key
            - name: REGISTRY_PROXY_USERNAME
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: proxy_username
            - name: REGISTRY_PROXY_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: proxy_password
          volumeMounts:
            - name: config-volume
              mountPath: /etc/distribution/
      volumes:
        - name: config-volume
          configMap:
            name: registry-config
            items:
            - key: config.yml
              path: config.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: registry-push
  name: registry-push
spec:
  selector: 
    matchLabels:
      app: registry-push
  template:
    metadata:
      labels:
        app: registry-push
      name: registry-push
    spec:
      containers:
        - image: "registry:3"
          name: registry
          ports:
            - containerPort: 5000
          env:
            - name: OTEL_TRACES_EXPORTER
              value: none
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
            - name: REGISTRY_STORAGE_S3_ACCESSKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: access_key
            - name: REGISTRY_STORAGE_S3_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: registry-storage-credentials
                  key: secret_key
          volumeMounts:
            - name: config-volume
              mountPath: /etc/distribution/
      volumes:
        - name: config-volume
          configMap:
            name: registry-config
            items:
            - key: config-push.yml
              path: config.yml

And finaly the httpRoute resource, the star of the night:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: registry
spec:
  parentRefs:
  - name: traefik-gateway # This is Traefik's gateway name.
    namespace: traefik # In traefik's namespace
    sectionName: websecure # and I specificaly refenerce the websecure gateway which is the https one since I will be using the registry under https.
  hostnames:
  - my.registry.domain.com
# POST PUT and PATCH methods where the only methods that cannot be run against a proxy enabled registry and so I route them as so.
  rules:
  - matches:
    - method: POST
    backendRefs:
    - name: registry-push
      port: 5000
  - matches:
    - method: PATCH
    backendRefs:
    - name: registry-push
      port: 5000
  - matches:
    - method: PUT
    backendRefs:
    - name: registry-push
      port: 5000
# All other requests will be routed to the proxy enabled registry.
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: registry
      port: 5000

Lets test it.

First lets do some pull-through caching tests.

➜  ✗ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
registry-5d8b4c5b66-5ftvf        1/1     Running   0          3h15m
registry-push-6b694b46b9-wzw2d   1/1     Running   0          3h15m

➜  ✗ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
97e70d161e81: Pull complete 
Digest: sha256:37f7b378a29ceb4c551b1b5582e27747b855bbfaa73fa11914fe0df028dc581f
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

➜  ✗ kubectl logs registry-5d8b4c5b66-5ftvf
"GET /v2/library/busybox/blobs/sha256:97e70d161e81def43e2a371dea30a2ceb2e226e657cac20a243224f21c1bb36f HTTP/1.1" 200 2145250 "" "docker/28.0.4 go/go1.23.7 git-commit/6430e49 kernel/6.13.9-200.fc41.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.4 \\(linux\\))"

Checking the minio console validates that a busybox image is now present in my docker-registry bucket. Now lets try and push some images.

➜ ✗ docker tag tensorflow/tensorflow:latest my.registry.domain.com/tensorflow/tensorflow:latest
➜ ✗ docker push my.registry.domain.com/tensorflow/tensorflow:latest
The push refers to repository [my.registry.domain.com/tensorflow/tensorflow]
e6fa850fd8cc: Layer already exists 
b70b811e5676: Layer already exists 
1dcaeee65195: Layer already exists 
6fa0e3e95d00: Layer already exists 
631cd10af51c: Layer already exists 
183e7a7af206: Layer already exists 
f4cd9bdd92f2: Layer already exists 
f8c78b408f5d: Layer already exists 
2495320ee7f6: Layer already exists 
44390fb12d12: Layer already exists 
496acd2c9151: Layer already exists 
270a1170e7e3: Layer already exists 
latest: digest: sha256:f24e8494d43a4e613edd7fbd3782594368d52d79af1e216051139ed2d7830682 size: 2829

➜ ✗  kubectl logs registry-push-6b694b46b9-wzw2d
level=info msg="response completed" go.version=go1.23.7 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=my.registry.domain.com http.request.id=eada32b2-b3dc-4a45-8f4f-23bb8b168161 http.request.method=PUT http.request.remoteaddr=172.30.1.1 http.request.uri=/v2/tensorflow/tensorflow/manifests/latest http.request.useragent="docker/28.0.4 go/go1.23.7 git-commit/6430e49 kernel/6.13.9-200.fc41.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.4 \\(linux\\))" http.response.duration=288.116854ms http.response.status=201 http.response.written=0 instance.id=e9491b4c-a84a-4eb2-aa40-4a365bcbac57 service=registry vars.name=tensorflow/tensorflow vars.reference=latest version=3.0.0
"PUT /v2/tensorflow/tensorflow/manifests/latest HTTP/1.1" 201 0 "" "docker/28.0.4 go/go1.23.7 git-commit/6430e49 kernel/6.13.9-200.fc41.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.4 \\(linux\\))"

The PUT request was only routed to the push registry while the rest of the requests were routed to the other registry.

FIN

So thats was it! Now I can sleep in peace knowing that no more the world will need to go with complex registry installations that require backup and maintenance, in order to have the 2 very basic features a registry should support out of the box. Thank you for reading! Bye! P.S: Fun fact this blog was build and pushed to this registry.

parask.me