Sockets in /tmp deleted by features/docker-in-docker

[rhettg]

This is a more general case of #860 that is worse, but trickier to reproduce.

After adding a new feature, tailscale, my SSH Agent stopped working. The environment variable SSH_AUTH_SOCK was set, but it pointed to a socket in /tmp that no longer existed. It turns out to be a bad interaction between some features and docker-in-docker.

The interaction is order dependent. It can be avoided by explicitly ensuring docker-in-docker is installed first (or at least early) by specifying overrideFeatureInstallOrder.

This seems to be triggered by how docker-in-docker creates a tmpfs /tmp as part of it's entrypoint:

features/src/docker-in-docker/install.sh

Lines 534 to 537 in 31f99a0

	# Mount /tmp (conditionally)
	if ! mountpoint -q /tmp; then
	mount -t tmpfs none /tmp
	fi

When docker-in-docker replaces /tmp, it very predictably drops anything that was already in /tmp. This is always the case. What happens with the interaction with the tailscale feature is that if tailscale installs first, then our sockets are created in the /tmp that is replaced by the later entrypoint of docker-in-docker. This appears to be a race!

I've been able to minimally reproduce this by creating a noop feature that simply sleeps for 30 seconds during it's entrypoint. When it runs first, sleeping for 30 seconds, then the docker-in-docker entrypoint runs removing our sockets.

https://github.com/rhettg/dind-feature-bug

[Kaniska244]

Hello @rhettg ,

My sincere apologies for the delay. Mounting the /tmp as tmpfs is required for the D-in-D feature (mainly the moby-cli) and doesn't appear to be a wise choice to remove it right away. Would request you to continue using the overrideFeatureInstallOrder in the dev container configuration to force the feature installation order during the build.
Kindly let me know in case of further concern on this.

[rhettg]

No worries, I'm not blocked, but it's obviously a bit of a foot-gun.

Suggestions off the top of my head:

1. Could devcontainers always have a tmpfs?
2. Could d-in-d fail-fast if tmpfs isn't available?
3. Could d-in-d log / warn about overwriting sockets in /tmp?

[zacaj]

#3 would be very appreciated at a minimum. Just spent about three hours narrowing down the cause of my ssh agent forwarding not working to this (plus desktop-lite) and there isn't any indication in the logs/etc that anything is wrong, or even that d-in-d is touching /tmp at all

I'd go even further to say that the container build should just fail if d-in-d detects that it's not the first feature installing / it's going to overwrite folders in /tmp / etc

Not sure why it is that if you have just d-in-d but not another feature that vs-code's ssh agent forwarding still works, but adding a second feature breaks it, even though it's not related to that second feature...