K3s Docker pull through registry
A cluster local Docker pull through registry can speedup your deployments. The registry checks if an image exists locally, if not, it pulls it from dockerhub and caches it locally.
You can apply (something like) the following Kubernetes deployment to run the registry in your cluster, and expose it on NodePort 30555. If you want to try this deployment, make sure you adapt the persistent volume claim for your cluster settings.
--- apiVersion: v1 kind: Namespace metadata: name: docker-registry --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: docker-registry-pvc namespace: docker-registry spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi storageClassName: managed-nfs-storage --- kind: Service apiVersion: v1 metadata: name: docker-registry namespace: docker-registry spec: type: NodePort ports: - port: 80 targetPort: http nodePort: 30555 selector: app: docker-registry --- kind: Deployment apiVersion: apps/v1 metadata: name: docker-registry namespace: docker-registry spec: replicas: 1 selector: matchLabels: app: docker-registry template: metadata: labels: app: docker-registry spec: containers: - name: docker-registry image: 'thomasdegraaff/registry-pull-through:2.7.1' # env: # - name: REGISTRY_USERNAME # value: user # - name: REGISTRY_PASSWORD # valueFrom: # secretKeyRef: # name: docker-registry-password # key: password ports: - name: http containerPort: 5000 protocol: TCP livenessProbe: httpGet: port: http initialDelaySeconds: 60 readinessProbe: httpGet: port: http initialDelaySeconds: 10 volumeMounts: - mountPath: /var/lib/registry name: docker-registry-data volumes: - name: docker-registry-data persistentVolumeClaim: claimName: docker-registry-pvc
The image used is based on the original Docker registry image, with the addition of a pull through registry config file. Optionally one can add environment variables when one needs to access a private dockerhub registry. See the Dockerfile for the build specs.
Once the registry is running, you can check if it is ok:
If everything is ok, the result should be something like this:
Now you need to configure k3s to use the pull through registry. On each node add the following file and restart k3s. Use the mirror name docker.io to overrule the default dockerhub the cluster uses.
Now deploy a test app to check if the cache is working.
--- apiVersion: v1 kind: Namespace metadata: name: learning --- apiVersion: v1 kind: Service metadata: namespace: learning name: hello-nodeport spec: type: ClusterIP ports: - protocol: TCP port: 80 selector: run: hello-world --- apiVersion: apps/v1 kind: Deployment metadata: namespace: learning name: hello-world spec: selector: matchLabels: run: hello-world replicas: 1 template: metadata: namespace: learning labels: run: hello-world spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80 protocol: TCP
Now check the registry again for repositories.
When the test fails, and you want to test again, make sure the nginxdemos/hello image is removed from the cluster. Remove the hello world deployment, and then remove the image on your cluster node(s).
I had some trouble debugging when testing because I thought that the setting
imagePullPolicy: Always ment that the image was always pulled from Dockerhub. But this is not true. When a image exists in the cluster node with the exact same digest as the current image on Dockerhub, then the local image is used.