Documentation Notice: `push from-args` is overwrite, not append.

[jnovack]

A major caveat on DockerHub (not your fault) of push from-args is that the operation overwrites what is there, not appends.

This means, you can not dynamically add additional architectures to a pre-existing list or do a multiple-builds and have them join up under 1 tag, you must have the complete list, and push it only at the end when all builds are complete.

I think this should be called out, in documentation, somewhere. I spent 4 hours wondering why my multi-architecture builds were not showing. 😠

/ # ./manifest-tool inspect jnovack/autossh:test
Name:   jnovack/autossh:test (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:3bf6448f1d1ab44c2691d37ad5272afc3364eaecb5b8c815098f9fd5bedd7ed9
 * Contains 1 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:876d2648e610fdf224246561d57bd340316ad73d9cf8c6b94c8f19935b66db46
1  Mfst Length: 1157
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: arm
1           - Variant:
1           - Feature:
1     # Layers: 4
         layer 1: digest = sha256:52278dd8e57993669c5b72a9620e89bebdc098f2af2379caaa8945f7403f77a2
         layer 2: digest = sha256:cd1a4a0564f1988e37a1cb0380397a2db9f28824ac7c1abdb855201bc72cccd0
         layer 3: digest = sha256:772672a29dadab958356b473b3f8487ecc6c681e142fb5b60f8e852282f13988
         layer 4: digest = sha256:40f3bdac7e4592aeb10960ab7bef50f66f1e92195aa0d26bcda2e543babed85f

/ # ./manifest-tool push from-args --platforms linux/arm64 --template jnovack/autossh:v2.0-arm64v8 --target jnovack/autossh:test
Digest: sha256:8285c7c23832ca45fe478d46376a4c56fd8392c213e9312ad0055b34c3da6b89 434

/ # ./manifest-tool inspect jnovack/autossh:test
Name:   jnovack/autossh:test (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:8285c7c23832ca45fe478d46376a4c56fd8392c213e9312ad0055b34c3da6b89
 * Contains 1 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:575c937126cbdef12e64b98366c6901c213e93b1e645fbeed0cb6369464b17d6
1  Mfst Length: 1157
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: arm64
1           - Variant:
1           - Feature:
1     # Layers: 4
         layer 1: digest = sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd
         layer 2: digest = sha256:e47f356fda3fcbd5ce6a215c9910b4dbf57ac7e423dffc227de544d1e74d6a28
         layer 3: digest = sha256:207cbbd7dc95f863704f1b187b77a6eeda0c82edcb0f491365c29da0e32b295b
         layer 4: digest = sha256:250a14336a497fb414bbda6e0c4882e6af6b4485e7784a64b159a6af4dcfbe3b

[jnovack]

The question of "Am I Doing It Wrong(tm)?" comes up.

My autobuild on Docker Hub looks like this:

And my post_push (which runs PER build) looks like this:

./manifest-tool push from-args \
    --platforms linux/"$TARGET_ARCH" \
    --template "$IMAGE_NAME" \
    --target "$DOCKER_REPO:${1}"

This, in effect, runs manifest-tool once for each build, and knows about one architecture (just the one being built.

./manifest-tool push from-args --platforms linux/arm/v6 --template jnovack/autossh:arm32v6 --target jnovack/autossh:latest

And then in the next build...:

./manifest-tool push from-args --platforms linux/arm/v7 --template jnovack/autossh:arm32v7 --target jnovack/autossh:latest

Which has the net result of overwriting, not appending. The only one listed is the last one which gets built.

Is there a more-correct way? OR Can push from-args APPEND rather than overwrite?

(Note: The reason I'm not doing a multi-arch-manifest.yml is to avoid hard-coding as much as possible.)

I can run the manifest push only if the DOCKER_FILE == Dockerfile, but there is no guarantee it will build last to properly get all the images in the manifest. Even still, I would have to hard code the values of the other architectures.

Finally, comes the newer practice of backfilling your versions. v2.1.3 updates :2.1.3, :2.1, and :2. This adds an entirely different dimension to the problem.

[jnovack]

Even still, I would have to hard code the values of the other architectures.

In a flint of inspiration, I realized that, "No, I don't have to hard code the other archs, I can find/glob the Dockerfiles.

Once I did that, I realized that I don't only need to build ONE container, I can build ALL THE THINGS. This way I can keep track of them.

Long story short (too late!), I'm now building all my architectures within one build in Docker Hub, and doing one push of ./manifest-tool at the end.

If you are curious about the code, a template repository is available at jnovack/docker-multi-arch-hooks.

[estesp]

These are some good questions, and I'm sure the documentation in this repo could be more clear on the usage side for various scenarios. Mostly the topic of "how do I?" has been shared in other venues like talks about multi-arch build pipelines and/or specific users (like how manifest-tool is used by the official images update scripts for DockerHub). I should probably add a section to link to some of those resources to give folks a jumping off point to learn some of the practices others are using.

To respond to a few specifics, although it sounds like you are getting most of this:

• the from-args usage makes the most sense when you have a set of images you have already built and used a very specific naming structure such that you can use the template with replacement values for OS, ARCH, and VARIANT and manifest-tool does the work of going and looking up those image references and assembling the final multi-platform image specified as the target.
• There is no real way to "append" to a manifest-list; image references and content changing the manifest JSON will create a new image with a new cryptographic hash/identifier, so just like rebuilding a (non-multiarch) image and pushing it to the same tag overwrites what the registry already has for that tag, the same thing happens when writing a manifest list object to a registry.
• To have the semblance of a growing manifest list entry as architecture builds complete requires using the YAML file input option alongside the "--ignore-missing" option so that you can keep doing manifest-list push after each build completes with the same YAML listing all images you expect to be part of the final manifest list, and it will keep writing it with the ones which are complete and ignoring (with a WARN output) ones which are missing. The official image build tools use this model so that, for example, "busybox:latest" will have as many architectures are done, and will update as often as necessary until the last architecture build is complete, which could be hours after the first one completes.
• The same official images team (and @tianon can chime in here if he has other details) also asked for the feature within the YAML input to list an additional set of tags you want to update to also point to the manifest list object, so you can update "latest" and "2.3.1" and "2.3" and "2" all at the same time. This is not available on the command line from-args model, but only in the YAML input.

All my projects which use manifest-tool are using the model you noted in your last comment: you build all images in one go and then run manifest-tool to compile them into the manifest list object entry in the registry. I use that here in this project (you can see the CI configuration and how I do it), in my mquery tool repo, and also @StefanScherer has some good resources like this blog post on his model of build, push, then push multiplatform manifest. Note that he is using the experimental docker manifest command, which was written from the code here in manifest-tool.