Skip to content

K3s Docker pull through registry

A cluster local Docker pull through registry can speedup your deployments. The registry checks if an image exists locally, if not, it pulls it from dockerhub and caches it locally.

You can apply (something like) the following Kubernetes deployment to run the registry in your cluster, and expose it on NodePort 30555. If you want to try this deployment, make sure you adapt the persistent volume claim for your cluster settings.

docker-registry.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: docker-registry
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: docker-registry-pvc
  namespace: docker-registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: managed-nfs-storage
---
kind: Service
apiVersion: v1
metadata:
  name: docker-registry
  namespace: docker-registry
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: http
      nodePort: 30555
  selector:
    app: docker-registry
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: docker-registry
  namespace: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: 'thomasdegraaff/registry-pull-through:2.7.1'
#          env:
#            - name: REGISTRY_USERNAME
#              value: user
#            - name: REGISTRY_PASSWORD
#              valueFrom:
#                secretKeyRef:
#                  name: docker-registry-password
#                  key: password
          ports:
            - name: http
              containerPort: 5000
              protocol: TCP
          livenessProbe:
            httpGet:
              port: http
            initialDelaySeconds: 60
          readinessProbe:
            httpGet:
              port: http
            initialDelaySeconds: 10
          volumeMounts:
            - mountPath: /var/lib/registry
              name: docker-registry-data
      volumes:
        - name: docker-registry-data
          persistentVolumeClaim:
            claimName: docker-registry-pvc

The image used is based on the original Docker registry image, with the addition of a pull through registry config file. Optionally one can add environment variables when one needs to access a private dockerhub registry. See the Dockerfile for the build specs.

Once the registry is running, you can check if it is ok:

curl http://[a-cluster-node-ip]:30555/v2/_catalog

If everything is ok, the result should be something like this:

{"repositories":[]}

Now you need to configure k3s to use the pull through registry. On each node add the following file and restart k3s. Use the mirror name docker.io to overrule the default dockerhub the cluster uses.

/etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "http://127.0.0.1:30555"

Now deploy a test app to check if the cache is working.

hello-world.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: learning
---
apiVersion: v1
kind: Service
metadata:
  namespace: learning
  name: hello-nodeport
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
  selector:
    run: hello-world
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: learning
  name: hello-world
spec:
  selector:
    matchLabels:
      run: hello-world
  replicas: 1
  template:
    metadata:
      namespace: learning
      labels:
        run: hello-world
    spec:
      containers:
        - name: hello-world
          image: nginxdemos/hello:0.2
          ports:
            - containerPort: 80
              protocol: TCP

Now check the registry again for repositories.

$ curl http://[a-cluster-node-ip]:30555/v2/_catalog
If the output contains a nginxdemos/hello repo, success!
{"repositories":["nginxdemos/hello"]}
This means that the pull through registry has cached the nginxdemos/hello image.

When the test fails, and you want to test again, make sure the nginxdemos/hello image is removed from the cluster. Remove the hello world deployment, and then remove the image on your cluster node(s).

sudo k3s crictl rmi --prune

I had some trouble debugging when testing because I thought that the setting imagePullPolicy: Always ment that the image was always pulled from Dockerhub. But this is not true. When a image exists in the cluster node with the exact same digest as the current image on Dockerhub, then the local image is used.

Back to top