Mounting Secrets and ConfigMaps in a preexistent folder without deleting the folder's contents

I bet you've been there: you mounted a ConfigMap and nuked the entire preexistent directory!

We’ve all been there: we’re working on our next super-hyper-duper Kubernetes operator, we’re about to deploy it but we’re doing some local testing, so we create a ConfigMap or a Secret, we mount it to the Pod, launch our app and we see the entire directory is now gone, replaced with our ConfigMap or Secret’s contents.

This post will show you how to mount a ConfigMap or Secret on a preexistent folder without deleting all its data. We’ll use the hypothetical case scenario that you’re working in your next static, pure HTML page – and of course, you do need Kubernetes to run that because reasons – so you’ll resort to using the Nginx image.

Getting to know the Container image

We’ll use the following nginx image, pinned to a specific version in case this post becomes old 😉

docker pull nginx:1.23.2

The image is quite simple: it spins up an already configured Nginx server listening on port 80 and serving the contents of the /usr/share/nginx/html directory. Here, there’s a default HTML file called index.html and a 50x.html file that will be served if the server encounters an error:

root@nginx:/usr/share/nginx/html# ls
50x.html  index.html

Another cool thing about this Nginx container is that it automatically serves all the contents of this directory via port 80. If you were to run this container locally, you can access the index.html file as well as the 50x.html:

docker run -d -p 8080:80 nginx:1.23.2
b3a72263a159ca3a1547af706281920e745782eea8dc044bb0a9ad6b807318ab

curl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 22:50:37 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-267"
Accept-Ranges: bytes

curl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 22:51:14 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytes

If we add our files to the Nginx directory, they’ll be served by the Nginx server too, and we can access them if we know their name, so a myfile.log would be served via localhost:8080/myfile.log.

Now, for the sake of the example, we want to add files to the Nginx folder without deleting any file that’s already there – that includes both the index.html and the 50x.html. For our Kubernetes example, we will grab these from a Secret, but you can use a ConfigMap as well, by just changing certain fields in your YAML manifest.

Creating the Secret

Now let’s see the nuke-culprit: this is the Kubernetes Secret we’ll be using:

apiVersion: v1
kind: Secret
metadata:
  name: test-secrets
stringData:
  username: patrick
  password: covfefe
  "config.cfg": |
    [settings]
    enable-health-checks: true
    [server]
    port: 8080

We’re then creating 3 values inside a single Kubernetes Secret – and again, it could’ve been a ConfigMap but to make it more challenging and because we all love base64, we’ll use a Secret instead:

• username with the value patrick; and
• password with the value covfefe; and
• a config.cfg file with some made-up settings, but useful to demonstrate longer files mounted as well

The config.cfg also has a name that looks like a file you would find in your own computer as part of, say, your fancy Java app or something similar that works with cfg files… Or maybe it’s a toml with a wrong extension? 🤣

Now let’s mount these to the Nginx container.

Mounting the Secret’s contents as files

Mounting Kubernetes Secrets or ConfigMaps is quite straightforward, although there are quite a few settings you can tweak to make it work in different ways.

We also want to make sure we don’t delete the contents of the /usr/share/nginx/html directory too. We want our fancy HTML “hello, world!” page to be there, and we want to add our Secret’s contents to it.

The Secret

We will use the following ConfigMap for all the examples below:

apiVersion: v1
kind: Secret
metadata:
  name: test-secrets
stringData:
  username: patrick
  password: covfefe
  "config.cfg": |
    [settings]
    enable-health-checks: true
    [server]
    port: 8080

Apply this manifest and keep it in your cluster for the rest of the examples:

kubectl apply -f secret.yaml
secret/test-secrets created

The initial, naive approach

Let’s start with the naive approach: we’ll mount the Secret to the /usr/share/nginx/html directory, and we’ll see what happens. Use the following YAML, which includes the Secret and a made-up Pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.23.2
      volumeMounts:
        - name: test-volume
          mountPath: /usr/share/nginx/html/
          readOnly: true
  volumes:
    - name: test-volume
      secret:
        secretName: test-secrets

Note the highlighted lines: the first highlighted block states that we want to mount a volume called test-volume in the path /usr/share/nginx/html/. We define that volume in the second highlighted piece of code.

Let’s see what happens to the Nginx folder and its contents. Apply the manifest, making sure the previously defined Secret was already applied:

kubectl apply -f deployment.yaml
pod/nginx created

Then review the folder contents:

kubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
config.cfg  password  username

So what we see here is that there are 3 files: config.cfg, password and username. The index.html and the 50x.html files have disappeared: mounting “volumes” doesn’t work the same way in Kubernetes vs Docker. While in Docker you can specify mounting volumes in a particular way – even as a readonly option without overwriting existent files – in Kubernetes this is not the case.

In other words, we mounted the Secret but it ended up overwriting our files in the html folder with the values from the Secret and as such, we lost our index.html and our nifty Nginx error page, the 50x.html. Let’s try to fix that.

We can also see this using the HTTP server Nginx spins up, if we try to request the 50x.html file we are greeted with an error. Create a port forward to access the Nginx pod you just created. This assumes the pod landed in the default namespace:

kubectl port-forward pod/nginx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Then request the 50x.html file:

curl -I localhost:8080/50x.html
HTTP/1.1 404 Not Found
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 23:40:15 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

Same result if we try to request the index.html file:

curl -I localhost:8080/index.html
HTTP/1.1 404 Not Found
Server: nginx/1.23.2
Date: Mon, 31 Oct 2022 23:40:43 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

Approach 1: Using postStart hooks

The first solution is to use a postStart hook. This is a command that runs after the container has started, and it’s a good way to “fix” the contents of the mounted volume. To do this, however, you have to:

• Mount the contents of the Secret in a different directory than the one you want to use; and
• Move these files – or symlink them – once the container has started

One important thing to note about postStart is that Kubernetes makes no guarantees postStart code will run before your container’s ENTRYPOINT (or the command field in your YAML code). This could mean your postStart code might move the secret contents after your container has started. If your code doesn’t care or already accounts for this “delay” then this solution is acceptable.

The second caveat is that, well, your code now is far more verbose and kinda looks ugly 😅 but if that doesn’t bother you as much as it bothers me, then go right ahead! 🤣

The following example uses the postStart approach:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.23.2
      volumeMounts:
        - name: test-volume
          mountPath: /tmp/secrets/
          readOnly: true
      lifecycle:
        postStart:
          exec:
            command:
              - "/bin/sh"
              - "-c"
              - "cp /tmp/secrets/* /usr/share/nginx/html/"
  volumes:
    - name: test-volume
      secret:
        secretName: test-secrets

If we delete and re-apply the manifest, we can see that the index.html and 50x.html files are back:

kubectl delete -f pod.yaml && kubectl apply -f pod.yaml
pod "nginx" deleted
pod/nginx created

kubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
50x.html  config.cfg  index.html  password  username

Now we see our usual suspects: the index.html and 50x.html files from Nginx are still there, and the config.cfg, password and username files are also there. Good! Still, keep in mind the caveats of this approach considering there are no guarantees the postStart code will run before your own application’s code. If you’re using this to configure, say, a database’s password for example, your Pod will crash.

With this method though, our usual suspects still work. Port-forward the Pod again with kubectl port-forward pod/nginx 8080:80 and request the 50x.html file, for example:

curl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:22:39 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytes

Same thing happens with our secret-turned-into-file:

curl -I localhost:8080/config.cfg
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:23:03 GMT
Content-Type: application/octet-stream
Content-Length: 58
Last-Modified: Tue, 01 Nov 2022 00:16:15 GMT
Connection: keep-alive
ETag: "6360654f-3a"
Accept-Ranges: bytes

But in my personal opinion, this is good, but not enough. The lack of guarantee the files will be there before my application starts make it a no-no for certain scenarios.

Approach 2: use subPath

In my personal opinion, the most appropriate solution here is to use Kubernetes’ own subPath parameter in your volumeMount declaration.

subPath is quite simple, but changes how your declaration of the mount looks like. For example, from our original attempt, the volumeMounts section looked like this:

volumeMounts:
  - name: test-volume
    mountPath: /usr/share/nginx/html/
    readOnly: true

The issue with this one was that it would delete the contents of the html folder and replace it with all our secrets instead. We can add to this declaration a subPath parameter, like this:

volumeMounts:
  - name: test-volume
    mountPath: /usr/share/nginx/html/config.cfg
    subPath: "config.cfg"
    readOnly: true

Now you might be about to say but Patrick, this example is way more verbose and now it only seems to mount a single Secret value! and you would be correct: the caveat of this method is that you’re no longer able to grab all secret values from an individual secret and instead, you have to reference them one by one.

Yes, it’s still quite verbose especially if you need to pull multiple secrets at once since you will have to repeat additional items in the array just to make it work… The benefits though? No more bash scripting on a postStart hook and the guarantee your files will be there before your application starts.

While it’s true it becomes quite verbose, I’m personally a fan of being more declarative rather than getting “Kubernetes Magic™️”: any new value registered against the Secret is now automatically available to the application for free with no changes. In my book? The verbosity and expressiveness here are a plus.

Let’s see it in action, let’s mount all 3 files:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.23.2
      volumeMounts:
        - name: test-volume
          mountPath: /usr/share/nginx/html/config.cfg
          subPath: "config.cfg"
          readOnly: true
        - name: test-volume
          mountPath: /usr/share/nginx/html/username
          subPath: "username"
          readOnly: true
        - name: test-volume
          mountPath: /usr/share/nginx/html/password
          subPath: "password"
          readOnly: true
  volumes:
    - name: test-volume
      secret:
        secretName: test-secrets

Once applied to the cluster we can see the files in the directory, no postStart required:

kubectl exec -it nginx -- bash -c "ls /usr/share/nginx/html/"
50x.html  config.cfg  index.html  password  username

And our usual suspects are still in place, even for HTTP requests – given you have restarted the port-forward in your machine:

curl -I localhost:8080/50x.html
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:36:52 GMT
Content-Type: text/html
Content-Length: 497
Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
Connection: keep-alive
ETag: "634fada5-1f1"
Accept-Ranges: bytes

And the config.cfg file:

curl -I localhost:8080/config.cfg
HTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Tue, 01 Nov 2022 00:37:14 GMT
Content-Type: application/octet-stream
Content-Length: 58
Last-Modified: Tue, 01 Nov 2022 00:34:58 GMT
Connection: keep-alive
ETag: "636069b2-3a"
Accept-Ranges: bytes

And just to prove the contents match what we set in the Secret, we can pull the content of these files too without specifying the -I flag for cURL:

curl localhost:8080/config.cfg
[settings]
enable-health-checks: true
[server]
port: 8080

curl localhost:8080/username
patrick

curl localhost:8080/password
covfefe

Among the caveats of this solution, besides its verbosity, are:

• It’s not possible to magically pull any new value inside your Secret automatically if you add new fields to your secret. If you want a new Secret value, you will have to update your Pod definition and redeploy it.
• subPath has some interesting behaviour: by default, without subpath, when you mount a Secret or ConfigMap into a folder, you get symbolic links (symlinks). These are used to dynamically update the value of the mounted files when the ConfigMap changes (no need for something like stackater/reloader), although your application will have to be able to handle this potential file content drift.

There you have it! 2 options to mount Secrets or ConfigMaps as files without deleting the folder they’re supposed to go in, in your Kubernetes Pods. Do you use any other trick to achieve the same result? Leave it in the comments below or ping me on Twitter! Happy to link your solution!