We’ve all been there: we’re working on our next super-hyper-duper Kubernetes operator, we’re about to deploy it but we’re doing some local testing, so we create a ConfigMap or a Secret, we mount it to the Pod, launch our app and we see the entire directory is now gone, replaced with our ConfigMap or Secret’s contents.
This post will show you how to mount a ConfigMap or Secret on a preexistent folder without deleting all its data. We’ll use the hypothetical case scenario that you’re working in your next static, pure HTML page – and of course, you do need Kubernetes to run that because reasons – so you’ll resort to using the Nginx image.
ConfigMap or Secret to a Pod, you can check out this article by tutorials.guide.We’ll use the following nginx image, pinned to a specific version in case this post becomes old 😉
docker pull nginx:1.23.2The image is quite simple: it spins up an already configured Nginx server listening on port 80 and serving the contents of the /usr/share/nginx/html directory. Here, there’s a default HTML file called index.html and a 50x.html file that will be served if the server encounters an error:
root@nginx:/usr/share/nginx/html# ls
50x.html index.htmlAnother cool thing about this Nginx container is that it automatically serves all the contents of this directory via port 80. If you were to run this container locally, you can access the index.html file as well as the 50x.html:
docker run -d -p 8080:80 nginx:1.23.2
b3a72263a159ca3a1547af706281920e745782eea8dc044bb0a9ad6b807318abHEAD request here to see if the files are there. If they were not available, we would get an HTTP 404 Not Found status code. A 200 OK means the file is available to be served via Nginxcurl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 22:50:37 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-267"
Accept-Ranges: bytescurl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 22:51:14 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytesIf we add our files to the Nginx directory, they’ll be served by the Nginx server too, and we can access them if we know their name, so a myfile.log would be served via localhost:8080/myfile.log.
Now, for the sake of the example, we want to add files to the Nginx folder without deleting any file that’s already there – that includes both the index.html and the 50x.html. For our Kubernetes example, we will grab these from a Secret, but you can use a ConfigMap as well, by just changing certain fields in your YAML manifest.
Now let’s see the nuke-culprit: this is the Kubernetes Secret we’ll be using:
apiVersion: v1
kind: Secret
metadata:
name: test-secrets
stringData:
username: patrick
password: covfefe
"config.cfg": |
[settings]
enable-health-checks: true
[server]
port: 8080 We’re then creating 3 values inside a single Kubernetes Secret – and again, it could’ve been a ConfigMap but to make it more challenging and because we all love base64, we’ll use a Secret instead:
username with the value patrick; andpassword with the value covfefe; andconfig.cfg file with some made-up settings, but useful to demonstrate longer files mounted as wellThe config.cfg also has a name that looks like a file you would find in your own computer as part of, say, your fancy Java app or something similar that works with cfg files… Or maybe it’s a toml with a wrong extension? 🤣
Now let’s mount these to the Nginx container.
Mounting Kubernetes Secrets or ConfigMaps is quite straightforward, although there are quite a few settings you can tweak to make it work in different ways.
We also want to make sure we don’t delete the contents of the /usr/share/nginx/html directory too. We want our fancy HTML “hello, world!” page to be there, and we want to add our Secret’s contents to it.
Secret We will use the following ConfigMap for all the examples below:
apiVersion: v1
kind: Secret
metadata:
name: test-secrets
stringData:
username: patrick
password: covfefe
"config.cfg": |
[settings]
enable-health-checks: true
[server]
port: 8080 Apply this manifest and keep it in your cluster for the rest of the examples:
kubectl apply -f secret.yaml
secret/test-secrets createdLet’s start with the naive approach: we’ll mount the Secret to the /usr/share/nginx/html directory, and we’ll see what happens. Use the following YAML, which includes the Secret and a made-up Pod:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.23.2
volumeMounts:
- name: test-volume
mountPath: /usr/share/nginx/html/
readOnly: true
volumes:
- name: test-volume
secret:
secretName: test-secretsNote the highlighted lines: the first highlighted block states that we want to mount a volume called test-volume in the path /usr/share/nginx/html/. We define that volume in the second highlighted piece of code.
Let’s see what happens to the Nginx folder and its contents. Apply the manifest, making sure the previously defined Secret was already applied:
kubectl apply -f deployment.yaml
pod/nginx createdThen review the folder contents:
kubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
config.cfg password usernameSo what we see here is that there are 3 files: config.cfg, password and username. The index.html and the 50x.html files have disappeared: mounting “volumes” doesn’t work the same way in Kubernetes vs Docker. While in Docker you can specify mounting volumes in a particular way – even as a readonly option without overwriting existent files – in Kubernetes this is not the case.
In other words, we mounted the Secret but it ended up overwriting our files in the html folder with the values from the Secret and as such, we lost our index.html and our nifty Nginx error page, the 50x.html. Let’s try to fix that.
We can also see this using the HTTP server Nginx spins up, if we try to request the 50x.html file we are greeted with an error. Create a port forward to access the Nginx pod you just created. This assumes the pod landed in the default namespace:
kubectl port-forward pod/nginx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80Then request the 50x.html file:
curl -I localhost:8080/50x.html
HTTP/1.1 404 Not Found
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 23:40:15 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-aliveSame result if we try to request the index.html file:
curl -I localhost:8080/index.html
HTTP/1.1 404 Not Found
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 23:40:43 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alivepostStart hooks The first solution is to use a postStart hook. This is a command that runs after the container has started, and it’s a good way to “fix” the contents of the mounted volume. To do this, however, you have to:
Secret in a different directory than the one you want to use; andOne important thing to note about postStart is that Kubernetes makes no guarantees postStart code will run before your container’s ENTRYPOINT (or the command field in your YAML code). This could mean your postStart code might move the secret contents after your container has started. If your code doesn’t care or already accounts for this “delay” then this solution is acceptable.
The second caveat is that, well, your code now is far more verbose and kinda looks ugly 😅 but if that doesn’t bother you as much as it bothers me, then go right ahead! 🤣
The following example uses the postStart approach:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.23.2
volumeMounts:
- name: test-volume
mountPath: /tmp/secrets/
readOnly: true
lifecycle:
postStart:
exec:
command:
- "/bin/sh"
- "-c"
- "cp /tmp/secrets/* /usr/share/nginx/html/"
volumes:
- name: test-volume
secret:
secretName: test-secretsIf we delete and re-apply the manifest, we can see that the index.html and 50x.html files are back:
kubectl delete -f pod.yaml && kubectl apply -f pod.yaml
pod "nginx" deleted
pod/nginx createdkubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
50x.html config.cfg index.html password usernameNow we see our usual suspects: the index.html and 50x.html files from Nginx are still there, and the config.cfg, password and username files are also there. Good! Still, keep in mind the caveats of this approach considering there are no guarantees the postStart code will run before your own application’s code. If you’re using this to configure, say, a database’s password for example, your Pod will crash.
With this method though, our usual suspects still work. Port-forward the Pod again with kubectl port-forward pod/nginx 8080:80 and request the 50x.html file, for example:
curl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:22:39 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytesSame thing happens with our secret-turned-into-file:
curl -I localhost:8080/config.cfg
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:23:03 GMT
Content-Type: application/octet-stream
Content-Length: 58
Last-Modified: Tue, 01 Nov 2022 00:16:15 GMT
Connection: keep-alive
ETag: "6360654f-3a"
Accept-Ranges: bytesBut in my personal opinion, this is good, but not enough. The lack of guarantee the files will be there before my application starts make it a no-no for certain scenarios.
subPath In my personal opinion, the most appropriate solution here is to use Kubernetes’ own subPath parameter in your volumeMount declaration.
subPath is quite simple, but changes how your declaration of the mount looks like. For example, from our original attempt, the volumeMounts section looked like this:
volumeMounts:
- name: test-volume
mountPath: /usr/share/nginx/html/
readOnly: trueThe issue with this one was that it would delete the contents of the html folder and replace it with all our secrets instead. We can add to this declaration a subPath parameter, like this:
volumeMounts:
- name: test-volume
mountPath: /usr/share/nginx/html/config.cfg
subPath: "config.cfg"
readOnly: trueNow you might be about to say but Patrick, this example is way more verbose and now it only seems to mount a single Secret value! and you would be correct: the caveat of this method is that you’re no longer able to grab all secret values from an individual secret and instead, you have to reference them one by one.
Yes, it’s still quite verbose especially if you need to pull multiple secrets at once since you will have to repeat additional items in the array just to make it work… The benefits though? No more bash scripting on a postStart hook and the guarantee your files will be there before your application starts.
While it’s true it becomes quite verbose, I’m personally a fan of being more declarative rather than getting “Kubernetes Magic™️”: any new value registered against the Secret is now automatically available to the application for free with no changes. In my book? The verbosity and expressiveness here are a plus.
Let’s see it in action, let’s mount all 3 files:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.23.2
volumeMounts:
- name: test-volume
mountPath: /usr/share/nginx/html/config.cfg
subPath: "config.cfg"
readOnly: true
- name: test-volume
mountPath: /usr/share/nginx/html/username
subPath: "username"
readOnly: true
- name: test-volume
mountPath: /usr/share/nginx/html/password
subPath: "password"
readOnly: true
volumes:
- name: test-volume
secret:
secretName: test-secretsOnce applied to the cluster we can see the files in the directory, no postStart required:
kubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
50x.html config.cfg index.html password usernameAnd our usual suspects are still in place, even for HTTP requests – given you have restarted the port-forward in your machine:
curl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:36:52 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytesAnd the config.cfg file:
curl -I localhost:8080/config.cfg
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:37:14 GMT
Content-Type: application/octet-stream
Content-Length: 58
Last-Modified: Tue, 01 Nov 2022 00:34:58 GMT
Connection: keep-alive
ETag: "636069b2-3a"
Accept-Ranges: bytesAnd just to prove the contents match what we set in the Secret, we can pull the content of these files too without specifying the -I flag for cURL:
curl localhost:8080/config.cfg
[settings]
enable-health-checks: true
[server]
port: 8080curl localhost:8080/username
patrickcurl localhost:8080/password
covfefeAmong the caveats of this solution, besides its verbosity, are:
Secret automatically if you add new fields to your secret. If you want a new Secret value, you will have to update your Pod definition and redeploy it.subPath has some interesting behaviour: by default, without subpath, when you mount a Secret or ConfigMap into a folder, you get symbolic links (symlinks). These are used to dynamically update the value of the mounted files when the ConfigMap changes (no need for something like stackater/reloader), although your application will have to be able to handle this potential file content drift.There you have it! 2 options to mount Secrets or ConfigMaps as files without deleting the folder they’re supposed to go in, in your Kubernetes Pods. Do you use any other trick to achieve the same result? Leave it in the comments below or ping me on Twitter! Happy to link your solution!