Tuesday, November 29, 2022

Docker "exec format error"

Overview

Occasionally, for a variety of reasons, a Docker container won't start, and the only clue is a cryptic message that says something like this:

standard_init_linux.go:178: exec user process caused "exec format error"

I'm aware that there are numerous related forum posts to be found via web search with a collection of right, wrong, confusing, and sometimes tediously wordy responses.  This is just an attempt to cut through some of that and cover only the non-wild-goose-chase stuff, not to take credit for any of it.  If any of this ended up worded similarly to something else, it's purely coincidence.

Cause #1 - Missing "shebang" in Shell Script

Sometimes, this is related to an improperly authored shell script, serving as the entrypoint for the container image, and the solution is to add the required "shebang" (#!/bin/sh) line at the top of the shell script.

Troubleshooting

  1. Run the image as a "shell" container and examine the entrypoint script 
    docker run --rm -it --entrypoint [] myimagename:myimagetag /bin/sh
  2. (Possible) Solution: Rebuild the container image with a corrected script

Cause #2 - Image/Runtime Platform Mismatch

The other main reason for the "exec format error" is that the platform for which the container image was built does not match the platform where the container is running.

This could be related to how the image was fetched / staged.  For instance, using "docker pull" from a public registry and "docker push" to a private registry, using a Macbook with an M1 processor (arm64 platform) when the container will be run in a kubernetes system that is on amd64 (intel compatible) hardware and/or virtual machines.

  • Note: When the docker image entrypoint references a binary executable (not a startup shell script... i.e. no chance of missing "#!/bin/sh"), it is more likely that the "platform mismatch" is the cause of the "exec format error".
  • Note: If the source registry does not have an image matching the client/target platform (e.g. on a Raspberry Pi Arm 32bit OS), docker pull might fetch an image with a "best match" platform, which won't work (wrong exec format).  This is the toughest one to fix, since it might require a full recompile of some binary executable files and probably demands building a new image up from multiple layers down.

Troubleshooting

  1. Check the image platform
    docker image inspect myimagename:myimagetag |grep -i architecture
  2. Check the runtime platform
    docker version |grep -i arch
    or
    kubectl version |grep -i server
  3. (Possible) Solutions
    1. Pull the image with the platform that matches the target runtime environment
      docker pull --platform linux/amd64 myimagename:myimagetag
      ...then push again to the target registry to overwrite/replace the wrong-platform image.
      See: https://docs.docker.com/build/building/multi-platform/
    2. Rebuild the image starting from the correct base image
      Modify Dockerfile and add --platform argument to the FROM line.
      See: https://docs.docker.com/engine/reference/builder/#from