+ + + + + +

+ +

+ So what is bootc? +

+ + +

+ + + +

+ +

About bootc

So what is bootc?

Imagine the ability to set up your operating system using a file you can check into source control. bootc basically turns your operating system into a containerized system, complete with in-place updates, custom configurations and pre-installed software and drivers, and more. You can share your custom containerized operating system image with others, too.

Unlock the power of bootc with shared custom OS images.

+ + +

+ + + + +

+ + + + + + + +

+ bootc Adopters (direct) +

+ via ostree) +

+ + +

End User Adopters

+
Note +Do you want to add yourself to this list? Simply fork the repository and open a PR with the required change. +We have a short description of the adopter types at the bottom of this page. Each type is in alphabetical order.
+

bootc Adopters (direct)

+ + + +

Type	Name	Since	Website	Use-Case
Vendor	Red Hat	2024	https://redhat.com	Image Based Linux
Vendor	HeliumOS	2024	https://www.heliumos.org/	An atomic desktop operating system for your devices

bootc Adopters (indirect, via ostree)

Bootc is a relatively new project, but much of the underlying technology and goals is not new. +The underlying ostree project is over 13 years old (as of 2024). This project also relates +to rpm-ostree which has existed a really long time.

Not every one of these projects uses bootc directly today, but a toplevel goal of bootc +is to be the successor to ostree, and it is our aim to seamlessly carry forward these users.

+ + + + + + + + +

Type	Name	Since	Website	Use-Case
Vendor	Endless	2014	link	A Completely Free, User-Friendly Operating System Packed with Educational Tools, Games, and More
Vendor	Red Hat	2015	link	Image Based Linux
Vendor	Apertis	2020	link	Collaborative OS platform for products
Vendor	Fedora Project	2021	link	An atomic desktop operating system aimed at good support for container-focused workflows
Vendor	Playtron GameOS	2022	link	A video game console OS that has integration with the top PC game stores
Vendor	Universal Blue	2022	link	The reliability of a Chromebook, but with the flexibility and power of a traditional Linux desktop
Vendor	Fyra Labs	2024	link	Bootc powers an experimental variant of Ultramarine Linux

Adopter Types

End-user: The organization runs bootc in production in some way.

Integration: The organization has a product that integrates with bootc, but does not contain bootc.

Vendor: The organization packages bootc in their product and sells it as part of their product.

+ + +

+ + + + +

+ + + + + +

+ +

Changelog

+ + +

+ + + + +

+ + + + + + + + + + + + + + + + + + + + + + + + + +

+ Contributing to bootc +

+ + - Note: Before writing a big patch + +

+ + - Development environment + +

+ + - Key recommended ingredients: + +

+ + - Ensure you're familiar with a bootc system + +

+ + - The Justfile + +

+ + - Code linting + +

+ + - Running the tests + +

+ + - Submitting a patch + +

+ + +

Contributing

Contributing to bootc

Thanks for your interest in contributing! At the current time, +bootc is implemented in Rust, and calls out to important components +which are written in Go (e.g. https://github.com/containers/image) +as well as C (e.g. https://github.com/ostreedev/ostree/). Depending +on what area you want to work on, you'll need to be familiar with +the relevant language.

Note: Before writing a big patch

If you plan to contribute a large change, please get in touch before +submitting a pull request by e.g. filing an issue describing your proposed +change. This will help ensure alignment.

Development environment

There isn't a single approach to working on bootc; however +the primary developers tend to use Linux host systems, +and test in Linux VMs. One specifically recommended +approach is to use toolbox +to create a containerized development environment +(it's possible, though not necessary to create the toolbox +dev environment using a bootc image as well).

At the current time most upstream developers use a Fedora derivative +as a base, and the hack/Containerfile defaults +to Fedora. However, bootc itself is not intended to strongly tie to a particular +OS or distribution, and patches to handle others are gratefully +accepted!

Key recommended ingredients:

A development environment (toolbox or a host) with a Rust and C compiler, etc. +While this isn't specific to bootc, you will find the experience of working on Rust +is greatly aided with use of e.g. rust-analyzer.
Install bcvk.

Ensure you're familiar with a bootc system

Worth stating: before you start diving into the code you should understand using +the system as a user and how it works. See the user documentation for that.

The Justfile

The Justfile is the primary interface for building and testing bootc.

just --list           # Show all targets organized by group
+just list-variants    # Show available build variants and current config
+

Building from source

Edit the source code; a simple thing to do is add e.g. +eprintln!("hello world"); into run_from_opt in crates/lib/src/cli.rs. +You can run make or cargo build to build that locally. However, a key +next step is to get that binary into a bootc container image.

Running just defaults to just build which will build a container +from the current source code; the result will be named localhost/bootc.

Running an interactive shell in an environment from the container

You can of course podman run --rm -ti localhost/bootc bash to get a shell, +and try running bootc.

Running container-oriented integration tests

just test-container
+

Running (TMT) integration tests

A common cycle here is you'll edit e.g. deploy.rs and want to run the +tests that perform an upgrade:

just test-tmt local-upgrade-reboot
+

To run a specific test:

just test-tmt readonly
+

Faster iteration cycles

The test cycle currently builds a disk image and creates a new ephemeral +VM for each test run.

You can shortcut some iteration cycles by having a more persistent +environment where you run bootc.

Upgrading from the container image

One good approach is to create a persistent target virtual machine via e.g. +bcvk libvirt run (or a cloud VM), and then after doing a just build and getting +a container image, you can directly upgrade to that image.

For the local case, check out cstor-dist. +Another alternative is mounting via virtiofs (see e.g. this PR to bcvk). +If you're using libvirt, see this document.

Running bootc against a live environment

If your development environment host is also a bootc system (e.g. a +workstation or a virtual server) one way to shortcut some cycles is just +to directly run the output of the built binary against your host.

Say for example your host is a Fedora 42 workstation (based on bootc), +then you can cargo b --release directly in a Fedora 42 container +or even on your host system, and then directly run e.g. ./target/release/bootc upgrade +etc.

Testing with composefs (sealed images)

To build and test with the experimental composefs backend:

# Build a sealed image with auto-generated test Secure Boot keys
+just build-sealed
+
+# Run composefs-specific tests
+just test-composefs
+
+# Validate that composefs digests match between build and install views
+# (useful for debugging mtime/metadata issues)
+just validate-composefs-digest
+

The build-sealed target generates test Secure Boot keys in target/test-secureboot/ +and builds a complete sealed image with UKI. See experimental-composefs.md +for more information on sealed images.

Debugging via lldb

The hack/lldb directory contains an example of how to use lldb to debug bootc code. +hack/lldb/deploy.sh can be used to build and deploy a bootc VM in libvirt with an lldb-server +running as a systemd service. Depending on your editor, you can then connect to the lldb server +to use an interactive debugger, and set up the editor to build and push the new binary to the VM. +hack/lldb/dap-example-vim.lua is an example for neovim.

The VM can be connected to via ssh test@bootc-lldb if you have nss +enabled.

For some bootc install commands, it's simpler to run the lldb-server in a container, e.g.

sudo podman run --pid=host --network=host --privileged --security-opt label=type:unconfined_t -v /var/lib/containers:/var/lib/containers -v /dev:/dev -v .:/output localhost/bootc-lldb lldb-server platform --listen "*:1234" --server
+

Code linting

The make validate target runs checks locally that we gate on +in CI, currently around cargo fmt and cargo clippy.

Running the tests

First, you can run many unit tests with cargo test.

container tests

There's a small set of tests which are designed to run inside a bootc container +and are built into the default container image:

$ just test-container
+

Submitting a patch

The podman project has some generic useful guidance; +like that project, a "Developer Certificate of Origin" is required.

Sign your PRs

The sign-off is a line at the end of the explanation for the patch. Your +signature certifies that you wrote the patch or otherwise have the right to pass +it on as an open-source patch. The rules are simple: if you can certify +the below (from developercertificate.org):

Developer Certificate of Origin
+Version 1.1
+
+Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+660 York Street, Suite 102,
+San Francisco, CA 94110 USA
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+

Then you just add a line to every git commit message:

Signed-off-by: Joe Smith <joe.smith@email.com>
+

Use your real name (sorry, no pseudonyms or anonymous contributions.)

If you set your user.name and user.email git configs, you can sign your +commit automatically with git commit -s.

Git commit style

Please look at git log and match the commit log style, which is very +similar to the +Linux kernel.

You may use Signed-off-by, but we're not requiring it.

General Commit Message Guidelines:

Title +
- Specify the context or category of the changes e.g. lib for library changes, docs for document changes, bin/<command-name> for command changes, etc.
- Begin the title with the first letter of the first word capitalized.
- Aim for less than 50 characters, otherwise 72 characters max.
- Do not end the title with a period.
- Use an imperative tone.
+
Body +
- Separate the body with a blank line after the title.
- Begin a paragraph with the first letter of the first word capitalized.
- Each paragraph should be formatted within 72 characters.
- Content should be about what was changed and why this change was made.
- If your commit fixes an issue, the commit message should end with Closes: #<number>.
+

Commit Message example:

<context>: Less than 50 characters for subject title
+
+A paragraph of the body should be within 72 characters.
+
+This paragraph is also less than 72 characters.
+

For more information see How to Write a Git Commit Message

+ + +

+ + + + +

+ + + + + + + + + + + + + + + +

+ Bootc Project Governance +

+ + - Values + +

+ + - Maintainers + +

+ + - Meetings + +

+ + - CNCF Resources + +

+ + - Code of Conduct + +

+ + - Security Response Team + +

+ + - Voting + +

+ + - Modifying this Charter + +

+ + +

Project governance

Bootc Project Governance

The Bootc project is dedicated to providing transactional, in-place operating system updates using OCI/Docker container images. +This governance explains how the project is run.

Values
Maintainers
Becoming a Maintainer
Meetings
CNCF Resources
Code of Conduct Enforcement
Security Response Team
Voting
Modifications

Values

The Bootc and its leadership embrace the following values:

+
Openness: Communication and decision-making happens in the open and is discoverable for future +reference. As much as possible, all discussions and work take place in public +forums and open repositories.
+
+
Fairness: All stakeholders have the opportunity to provide feedback and submit +contributions, which will be considered on their merits.
+
+
Community over Product or Company: Sustaining and growing our community takes +priority over shipping code or sponsors' organizational goals. Each +contributor participates in the project as an individual.
+
+
Inclusivity: We innovate through different perspectives and skill sets, which +can only be accomplished in a welcoming and respectful environment.
+
+
Participation: Responsibilities within the project are earned through +participation, and there is a clear path up the contributor ladder into leadership +positions.
+

Maintainers

Bootc Maintainers have "gated" write access to the project GitHub repository. +The current maintainers can be found in MAINTAINERS.md.

Direct pushes to the code is never allowed. All pull requests require review by a maintainer +other than the one submitting it. "Large" changes are encouraged to have a tracking +issue filed beforehand and gather consensus from multiple maintainers and interested community.

Maintainers collectively manage the project's resources and contributors.

This privilege is granted with some expectation of responsibility: maintainers +are people who care about the Bootc project and want to help it grow and +improve. A maintainer is not just someone who can make changes, but someone who +has demonstrated their ability to collaborate with the team, get the most +knowledgeable people to review code and docs, contribute high-quality code, and +follow through to fix issues (in code or tests).

A maintainer is a contributor to the project's success and a citizen helping +the project succeed.

The collective team of all Maintainers is known as the Maintainer Council, which +is the governing body for the project.

Becoming a Maintainer

To become a Maintainer you need to demonstrate the following:

commitment to the project: +
- participate in discussions, contributions, code and documentation reviews for 6 months or more,
- perform reviews for 20 non-trivial pull requests,
- contribute 10 non-trivial pull requests and have them merged,
+
ability to write quality code and/or documentation,
ability to collaborate with the team,
understanding of how the team works (policies, processes for testing and code review, etc),
understanding of the project's code base and coding and documentation style.

A new Maintainer must be proposed by an existing maintainer by opening a PR against the MAINTAINERS.md, which will prompt a gitvote. A simple majority vote of existing Maintainers +approves the application. Maintainers nominations will be evaluated without prejudice +to employer or demographics.

Maintainers who are selected will be granted the necessary GitHub rights.

Removing a Maintainer

Maintainers may resign at any time if they feel that they will not be able to +continue fulfilling their project duties.

Maintainers may also be removed after being inactive, failure to fulfill their +Maintainer responsibilities, violating the Code of Conduct, or other reasons. +Inactivity is defined as a period of very low or no activity in the project +for a year or more, with no definite schedule to return to full Maintainer +activity.

A Maintainer may be removed at any time by a 2/3 vote of the remaining maintainers.

Depending on the reason for removal, a Maintainer may be converted to Emeritus +status. Emeritus Maintainers will still be consulted on some project matters, +and can be rapidly returned to Maintainer status if their availability changes. +This requires two votes from active maintainers.

Meetings

Time zones permitting, Maintainers are expected to participate in the public +developer meeting, which occurs at 15:30 UTC on Fridays via Zoom.

Maintainers will also have closed meetings in order to discuss security reports +or Code of Conduct violations. Such meetings should be scheduled by any +Maintainer on receipt of a security issue or CoC report. All current Maintainers +must be invited to such closed meetings, except for any Maintainer who is +accused of a CoC violation.

CNCF Resources

Any Maintainer may suggest a request for CNCF resources, either in the +bootc discussions, or during a +meeting. A simple majority of Maintainers approves the request. The Maintainers +may also choose to delegate working with the CNCF to non-Maintainer community +members, who will then be added to the CNCF's Maintainer List +for that purpose.

Code of Conduct

Code of Conduct +violations by community members will be discussed and resolved +by the maintainers privately. If a Maintainer is directly involved +in the report, the Maintainers will instead designate two Maintainers to work +with the CNCF Code of Conduct Committee in resolving it.

Security Response Team

The Maintainers will appoint a Security Response Team to handle security reports. +This committee may simply consist of the Maintainer Council themselves. If this +responsibility is delegated, the Maintainers will appoint a team of at least two +contributors to handle it. The Maintainers will review who is assigned to this +at least once a year.

The Security Response Team is responsible for handling all reports of security +holes and breaches according to the security policy.

Voting

While most business in Bootc is conducted by "lazy consensus", +periodically the Maintainers may need to vote on specific actions or changes. +A vote can be taken using gitvote or +privately for security or conduct matters. Any Maintainer may demand a vote be taken.

Most votes require a simple majority of all Maintainers to succeed, except where +otherwise noted. Two-thirds majority votes mean at least two-thirds of all +existing maintainers.

Modifying this Charter

Changes to this Governance and its supporting documents may be approved by +a 2/3 vote of the Maintainers.

+ + +

+ + + + +

+ + + + + +

+ +

Project information

+
+ + Contributing + + +
+ +
+
+ + Maintainers + + +
+ +
+
+ + Adopters + + +
+ +
+
+ + About + + +
+ +
+
+ + Security + + +
+ +
+
+ + Changelog + + +
+ +
+
+ + Governance + + +
+ +

+ +

+ + + + +

+ + + + + + +

+ Maintainers +

+ Community Managers +

+ + +

Project maintainers

Maintainers

The current Maintainers Group for the bootc Project consists of:

+ + + + + + + +

Name	GitHub ID	Employer	Responsibilities
Chris Kyrouac	ckyrouac	Red Hat	Approver
Colin Walters	cgwalters	Red Hat	Approver
John Eckersberg	jeckersb	Red Hat	Approver
Xiaofeng Wang	henrywang	Red Hat	Approver
Gursewak Mangat	gursewak1997	Red Hat	Approver
Joseph Marrero	jmarrero	Red Hat	Approver

Community Managers

This group can represent the project for administrative and program manager duties. Examples: CNCF Service Desk tickets, coordinating with CNCF Project and Events teams, and LFX Administration. No code or code review rights.

+ + + +

Name	GitHub ID	Employer	Responsibilities
Laura Santamaria	nimbinatus	Red Hat	Representative
Mohan Shash	mohan-shash	Red Hat	Representative

+ + +

+ + + + +

+ + + + + + + + +

+ Security Policy +

+ + - Reporting a Vulnerability + +

+ + +

Security

Security Policy

Reporting a Vulnerability

If you find a potential security vulnerability in bootc, please report it by following these steps:

1. Use the GitHub Security Tab

This repository is set up to allow vulnerability reports through GitHub's Security Advisories feature. To report a vulnerability:

Navigate to the repository's main page.
Select the Security tab.
Select Advisories from the left-hand sidebar.
Click on Report a vulnerability.
Fill in the required details and submit the report.

Following this process will create a private advisory for our maintainers to review.

2. Do Not Open Public Pull Requests, Issues, or Discussions

Please do not discuss the issue, create PRs, or start discussions about the vulnerability. This ensures the vulnerability is not widely exploited before a fix is provided.

+ + +

+ + + + +

+ + + + + +

+ +

+ Containers: pitfalls of incomplete tar archives +

+ + +

+ - A refresher on tar +

+ +

+ - OCI image spec +

+ +

+ - Rechunking in bootc / rpm-ostree +

+ +

+ - Why isn't this a more widespread problem? +

+ +

+ - Overlayfs +

+ +

+ - Where do we go from here? +

+ +

+ - Conclusion +

+ + + +

+ +

+ December 15, 2025 + + by + + +John Eckersberg + + + + +

Containers: pitfalls of incomplete tar archives

As Colin Walters likes to say, containers +are just tarballs wrapped in JSON. We'll mostly ignore the JSON part +here, but there's a lot to unpack (pun and foreshadowing intended) +with regards to tar and how it interacts with the various pieces that +work in conjunction to make containers a reality.

A refresher on `tar`

Let's explore a key quirk of tar from which everything else we'll +discuss follows.

First, let's create a basic tar archive, which contains a directory +with a file inside of the directory:

⬢ [jeckersb@toolbx tmp]$ mkdir foo
+⬢ [jeckersb@toolbx tmp]$ touch foo/bar
+⬢ [jeckersb@toolbx tmp]$ tar cf foo.tar foo/
+

Now let's list the contents of the tar archive:

⬢ [jeckersb@toolbx tmp]$ tar tvf foo.tar
+drwxr-xr-x jeckersb/jeckersb 0 2025-12-10 11:09 foo/
+-rw-r--r-- jeckersb/jeckersb 0 2025-12-10 11:09 foo/bar
+

Ok, that makes sense. We see the directory and the file we added to +the archive.

But here's the key bit: What happens if you add only the file to the archive?

⬢ [jeckersb@toolbx tmp]$ tar cf incomplete.tar foo/bar
+⬢ [jeckersb@toolbx tmp]$ tar tvf incomplete.tar
+-rw-r--r-- jeckersb/jeckersb 0 2025-12-10 11:09 foo/bar
+

Now the archive has metadata only for the file. You can deduce from +the path foo/bar that there is a directory foo involved, but you +can't definitively know how foo is supposed to look. What are the +permissions, owner, group, mtime? We don't know. So what happens if +you extract this tar archive?

⬢ [jeckersb@toolbx tmp]$ mkdir incomplete-unpacked && tar xf incomplete.tar -C incomplete-unpacked
+⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked/
+total 0
+drwxr-xr-x. 2 jeckersb jeckersb 60 Dec 10 11:19 foo
+⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked/foo/
+total 0
+-rw-r--r--. 1 jeckersb jeckersb 0 Dec 10 11:09 bar
+

It implicitly creates the directory foo in order to put bar at the +correct path. This makes sense. But notice foo gets created with +metadata that is dependent on the environment in which the unpacking +of the archive happened. In the above example, you can see that the +timestamp on the directory is 10 minutes later than the timestamp of +the bar file inside. This is because it took me 10 minutes of +writing and editing this post before I did the unpack operation. I +can set my umask to something else and extract it a second time, and +now we'll see a different timestamp and different permissions:

⬢ [jeckersb@toolbx tmp]$ umask 0077
+⬢ [jeckersb@toolbx tmp]$ mkdir incomplete-unpacked-with-umask && tar xf incomplete.tar -C incomplete-unpacked-with-umask
+⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked-with-umask/
+total 0
+drwx------. 2 jeckersb jeckersb 60 Dec 10 11:25 foo
+⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked-with-umask/foo/
+total 0
+-rw-------. 1 jeckersb jeckersb 0 Dec 10 11:09 bar
+

OCI image spec

OCI container images are serialized as a series of +tarballs +that are applied on top of each other to create the complete +filesystem. We'll get back to the specifics of that operation a bit +later.

The entire layer specification is worth reading, but we'll focus on +two parts. First, how to determine changes:

When two directories are compared, the relative root is the top-level directory.
+The directories are compared, looking for files that have been [added, modified, or removed](#change-types).
+
+For this example, `rootfs-c9d-v1/` and `rootfs-c9d-v1.s1/` are recursively compared, each as relative root path.
+
+The following changeset is found:
+
+Added:      /etc/my-app.d/
+Added:      /etc/my-app.d/default.cfg
+Modified:   /bin/my-app-tools
+Deleted:    /etc/my-app-config
+
+This reflects the removal of `/etc/my-app-config` and creation of a file and directory at `/etc/my-app.d/default.cfg`.
+`/bin/my-app-tools` has also been replaced with an updated version.
+

And then the following section on representing changes:

A [tar archive][tar-archive] is then created which contains _only_ this changeset:
+
+- Added and modified files and directories in their entirety
+- Deleted files or directories marked with a [whiteout file](#whiteouts)
+
+The resulting tar archive for `rootfs-c9d-v1.s1` has the following entries:
+
+./etc/my-app.d/
+./etc/my-app.d/default.cfg
+./bin/my-app-tools
+./etc/.wh.my-app-config
+
+To signify that the resource `./etc/my-app-config` MUST be removed when the changeset is applied, the basename of the entry is prefixed with `.wh.`.
+

But note that the changeset says nothing about the directories higher +in the filesystem. So in the above example, it would be perfectly +valid (and more correct) to generate changeset tar archives which are +incomplete in the same manner we described above. Most notably, these +changesets would not contain tar entries for the directories /etc +and /bin.

This oversight in the specification is not new or surprising. There +is an open issue from +2017 which +describes exactly this problem. There is a linked pull +request which +has been open since late 2022 with intermittent discussion, but no +consensus or resolution has been reached yet.

This has led to some projects implementing their own workarounds to +deal with this behavior. The containerd project as an example +noted this +issue in 2017 +and worked around +it by creating +explicit archive directory entries for parents.

Rechunking in bootc / rpm-ostree

Base images used via bootc are typically post-processed and +finalized via the rpm-ostree +rechunker. +The general idea (oversimplifying a bit for brevity) is that the tool +can inspect the RPM database and map each file in the container to its +associated RPM. It then "rechunks" the container by creating a new +layer for each RPM, where the layer contains the files for that RPM +and only that RPM. This ensures that when a new version of an RPM is +released in the future, when it is integrated into a new base image, +only the layer associated with that RPM will need to be re-downloaded +by the end user.

However, consider a directory like /usr. This is owned by the +filesystem package which contains all of the shared base directories +in the system. This directory will be included in the layer +associated with the filesystem package with all of its related +metadata.

Now, almost every other package in the system is going to have some +amount of content that is shipped under the /usr hierarchy. +Originally, the rpm-ostree rechunker would omit the archive entry +for /usr in each package layer.

This however would cause the behavior of indeterminate tar archives +discussed above to be manifest in the final container. For example, +it was noted in this composefs-rs +issue.

We worked around this issue by creating a new +format-version for +the rpm-ostree rechunker, which creates all of the parent +directories in each tar layer. This is similar to the fix implemented +as noted above by the containerd project. This works, but it's not +ideal. It means that all of the parent directories and their +respective metadata have to be duplicated across every layer in the +image. This is an unnecessary waste of disk space and bandwidth.

Why isn't this a more widespread problem?

When I first realized what was going on here, I thought I could +trigger this behavior with a Containerfile like this:

FROM quay.io/fedora/fedora:43
+
+RUN <<EORUN
+
+# Create two levels of directories, because adding a file to a
+# directory alters the directory mtime.  This means adding a file
+# under /foo/bar in the next layer will cause the mtime for /foo/bar
+# to change, but /foo will remain unchanged.
+mkdir -p /foo/bar
+
+# zero the mtime on foo
+touch -d @0 /foo
+EORUN
+
+# Add a file in a derived layer
+#
+# In theory, this layer would not contain an entry for /foo, since it
+# is unchanged.
+#
+# In practice... not so much.
+RUN touch /foo/bar/baz
+

But alas:

⬢ [jeckersb@toolbx example]$ podman build -f Containerfile -t localhost/example
+STEP 1/3: FROM quay.io/fedora/fedora:43
+STEP 2/3: RUN <<EORUN (# Create two levels of directories, because adding a file to a...)
+--> d3f72d4d7b90
+STEP 3/3: RUN touch /foo/bar/baz
+COMMIT localhost/example
+--> b730a7409a5a
+Successfully tagged localhost/example:latest
+b730a7409a5a5b5450f4f1d142415043aeac48f6ac0d1ac2a632d75071668a60
+⬢ [jeckersb@toolbx example]$ podman run --rm -it localhost/example stat /foo
+  File: /foo
+  Size: 6               Blocks: 0          IO Block: 4096   directory
+Device: 0,93    Inode: 64851509    Links: 1
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Access: 2025-12-10 21:31:11.973931592 +0000
+Modify: 1970-01-01 00:00:00.000000000 +0000
+Change: 2025-12-10 21:31:11.973678196 +0000
+ Birth: 2025-12-10 21:31:11.973031361 +0000
+

But wait... the directory mtime is (correctly) still zeroed from the +first layer. So... everything up until now is a lie?

No! If we pull apart the last tar layer, we'll see that it actually +does contain a directory entry for /foo, even though it's not +modified in any way during the last RUN invocation. I'll skip the +part where I re-exported the image to an oci-dir just to make it +easier to look through the layers. Just trust me when I say this is +the tarball that gets generated for the last layer:

⬢ [jeckersb@toolbx sha256]$ tar tvf 9402e1d4a1daea10ff057116f7fc6858be340fd82204eb17db566bfec547a911.tar
+drwxr-xr-x 0/0               0 1969-12-31 19:00 foo/
+drwxr-xr-x 0/0               0 2025-12-10 16:31 foo/bar/
+-rw-r--r-- 0/0               0 2025-12-10 16:31 foo/bar/baz
+

Ok, so /foo is there. But... why? We didn't modify it. Deeper +down the rabbit hole we go.

Overlayfs

Spoiler: the answer is +overlayfs.

The OCI spec defines the image format, but notably it does not +specify any particular implementation details on how to store, manage, +and stitch together the final container image from its individual +layers.

In reality, container runtimes such as podman (via +containers-libs, +formerly containers-storage) use the kernel filesystem overlayfs to +stitch together the layers into the final realized view of the world.

A RUN invocation in a Containerfile is in a nutshell this series +of steps:

Mount the previous layer (and recursively, layers prior to that) +as a lowerdir in an overlayfs filesystem.
Perform the RUN invocation in the mounted filesystem.
Collect the contents of the resulting upperdir directory. This +becomes the tarball for that layer.

This process is how we end up with our /foo directory duplicated +into the final layer, even though we did not modify it. We can mock +this out just using an overlay mount and basic commands.

First, mock our lower layer, and set the mtime:

[root@burd tmp]# mkdir lower upper work merged
+[root@burd tmp]# mkdir -p lower/foo/bar
+[root@burd tmp]# touch -d @0 lower/foo
+[root@burd tmp]# stat lower/foo
+  File: lower/foo
+  Size: 60        	Blocks: 0          IO Block: 4096   directory
+Device: 0,45	Inode: 46858       Links: 3
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Context: unconfined_u:object_r:user_tmp_t:s0
+Access: 1969-12-31 19:00:00.000000000 -0500
+Modify: 1969-12-31 19:00:00.000000000 -0500
+Change: 2025-12-10 16:53:27.174443520 -0500
+ Birth: 2025-12-10 16:53:20.544455919 -0500
+

Next, mount it via overlayfs:

[root@burd tmp]# mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
+[root@burd tmp]# cd merged
+[root@burd merged]# stat foo
+  File: foo
+  Size: 60        	Blocks: 0          IO Block: 4096   directory
+Device: 0,90	Inode: 46858       Links: 3
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Context: unconfined_u:object_r:user_tmp_t:s0
+Access: 1969-12-31 19:00:00.000000000 -0500
+Modify: 1969-12-31 19:00:00.000000000 -0500
+Change: 2025-12-10 16:53:27.174443520 -0500
+ Birth: 2025-12-10 16:53:20.544455919 -0500
+

Now we simulate creating the new layer:

[root@burd merged]# touch foo/bar/baz
+

Finally we can unmount the overlayfs mount and observe the contents of +the upper dir:

[root@burd merged]# cd ..
+[root@burd tmp]# umount merged
+[root@burd tmp]# find upper/
+upper/
+upper/foo
+upper/foo/bar
+upper/foo/bar/baz
+[root@burd tmp]# stat upper/foo
+  File: upper/foo
+  Size: 60        	Blocks: 0          IO Block: 4096   directory
+Device: 0,45	Inode: 46870       Links: 3
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Context: unconfined_u:object_r:user_tmp_t:s0
+Access: 2025-12-10 16:57:20.828001765 -0500
+Modify: 1969-12-31 19:00:00.000000000 -0500
+Change: 2025-12-10 16:56:16.746593313 -0500
+ Birth: 2025-12-10 16:56:16.745126092 -0500
+

So that's where our /foo entry comes from. This behavior is +explained in the overlayfs +documentation:

Objects that are not directories (files, symlinks, device-special
+files etc.) are presented either from the upper or lower filesystem as
+appropriate. When a file in the lower filesystem is accessed in a way
+that requires write-access, such as opening for write access, changing
+some metadata etc., the file is first copied from the lower filesystem
+to the upper filesystem (copy_up). Note that creating a hard-link also
+requires copy_up, though of course creation of a symlink does not.
+
+The copy_up may turn out to be unnecessary, for example if the file is
+opened for read-write but the data is not modified.
+
+The copy_up process first makes sure that the containing directory
+exists in the upper filesystem - creating it and any parents as
+necessary. It then creates the object with the same metadata (owner,
+mode, mtime, symlink-target etc.) and then if the object is a file,
+the data is copied from the lower to the upper filesystem. Finally any
+extended attributes are copied up.
+

Since the content of the layer tarball is taken directly from the +upper, we end up getting a copy of /foo via this process.

Where do we go from here?

Hopefully I have explained the situation adequately and this all makes +sense. I have achieved some amount of inner peace that comes with +reaching understanding.

However... I still don't like it. I think we can do better.

Here's the thing for bootc: in our rechunked images, I don't want to +include the same directories over and over again in every single +layer. It shouldn't be necessary, and by strict reading of the OCI +spec we shouldn't do that if the directories are unchanged (which +they aren't).

But in the current state of the world, if we don't duplicate those +entries into every layer, we end up with non-deterministic directory +metadata when our OCI image is mounted and viewed through the lens of +overlayfs. This is fundamentally broken as we move bootc into a +future where +composefs and +fsverity are used +to harden and validate images. Ideally a container image should be +able to measure its own fsverity hash in a reproducible manner. Today +that is not possible.

So the following is what I've done as a proof-of-concept (a huge +motivation for writing this post in the first place was to +exhaustively explain my justification for why this is needed). I've +written and tested a patch (still needs refining before submitting) +for overlayfs which does the following:

Adds a new feature passthrough (naming bikeshedable) which +functions similarly to the metacopy feature.
Uses the trusted.overlay.passthrough to flag directories as +"structural-only".
When encountering a directory with the xattr, searches through the +lowerdirs until it finds a directory which does not have the xattr +set. The metadata for this layer is used. (If no unflagged +lowerdir is found, it falls back to the current behavior of using +non-deterministic local metadata)

Here's an example. This is similar to our manual overlayfs example +above, except this uses two lower layers to show how unpacking a tar +layer with incomplete directory information might look.

First, create our base layer again:

[root@fedora ~]# mkdir upper merged work
+[root@fedora ~]# mkdir -p base_layer/foo/bar
+[root@fedora ~]# touch -d @0 base_layer/foo
+

Then, in our second layer we create the baz file. This time we set +the passthrough xattr on /foo, which is how I envision this all +working when a container storage engine unpacks a tar archive with +incomplete parent directory information. Ideally this layer should +include a tar entry for foo/bar/baz and (arguably) foo/bar since +the directory metadata for bar should be updated to reflect the +addition of bar. But it definitely should not include an entry +for foo/:

[root@fedora ~]# mkdir -p second_layer/foo/bar
+[root@fedora ~]# touch second_layer/foo/bar/baz
+[root@fedora ~]# setfattr -n trusted.overlay.passthrough second_layer/foo
+

Now let's mount it with the patched overlayfs. First, we'll specify +passthrough=off to show the existing behavior:

[root@fedora ~]# mount -t overlay overlay -o upperdir=upper,lowerdir=second_layer:base_layer,workdir=work,passthrough=off merged
+[root@fedora ~]# stat merged/foo
+  File: merged/foo
+  Size: 6               Blocks: 0          IO Block: 4096   directory
+Device: 0,59    Inode: 365132      Links: 1
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Context: unconfined_u:object_r:admin_home_t:s0
+Access: 2025-12-10 17:33:32.787147889 -0500
+Modify: 2025-12-10 17:33:32.787147889 -0500
+Change: 2025-12-10 17:34:01.701054010 -0500
+ Birth: 2025-12-10 17:33:32.787147889 -0500
+

Here the foo directory has whatever metadata we created for it when +we "unpacked" it with the command mkdir -p second_layer/foo/bar +above. This is what we're trying to avoid.

Now let's unmount and remount with passthrough=on:

[root@fedora ~]# umount merged
+[root@fedora ~]# mount -t overlay overlay -o upperdir=upper,lowerdir=second_layer:base_layer,workdir=work,passthrough=on merged
+[root@fedora ~]# stat merged/foo
+  File: merged/foo
+  Size: 6               Blocks: 0          IO Block: 4096   directory
+Device: 0,59    Inode: 365129      Links: 1
+Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
+Context: unconfined_u:object_r:admin_home_t:s0
+Access: 1969-12-31 19:00:00.000000000 -0500
+Modify: 1969-12-31 19:00:00.000000000 -0500
+Change: 2025-12-10 17:28:12.268740241 -0500
+ Birth: 2025-12-10 17:27:39.158847877 -0500
+

Now we have the metadata for foo being passed through the second +layer and retrieved from the base layer, which is the behavior we +want.

Conclusion

Ensuring that OCI image layers are able to only ship content that +they've modified will aid reproducibility and help unlock +optimizations by avoiding having derived layers always need to inherit +metadata from their parent. At the current time, however, tooling +creating OCI images should include their parent information in order +to maximize compatibility. We are looking forward to a future where +that's not necessary though!

+ + +

+ + + + +

404 Not Found

So what is bootc?

bootc Adopters (direct)

bootc Adopters (indirect, via ostree)

Adopter Types

Contributing to bootc

Note: Before writing a big patch

Development environment

Key recommended ingredients:

Ensure you're familiar with a bootc system

The Justfile

Building from source

Running an interactive shell in an environment from the container

Running container-oriented integration tests

Running (TMT) integration tests

Faster iteration cycles

Upgrading from the container image

Running bootc against a live environment

Testing with composefs (sealed images)

Debugging via lldb

Code linting

Running the tests

container tests

Submitting a patch

Sign your PRs

Git commit style

Bootc Project Governance

Values

Maintainers

Becoming a Maintainer

Removing a Maintainer

Meetings

CNCF Resources

Code of Conduct

Security Response Team

Voting

Modifying this Charter

Maintainers

Community Managers

Security Policy

Reporting a Vulnerability

1. Use the GitHub Security Tab

2. Do Not Open Public Pull Requests, Issues, or Discussions

Containers: pitfalls of incomplete tar archives

A refresher on tar

OCI image spec

Rechunking in bootc / rpm-ostree

Why isn't this a more widespread problem?

Overlayfs

Where do we go from here?

Conclusion

+ Blog +

Archive

+ Your OS, but Containers! +

+ bootc builds operating systems using containerization principles +

Motivation

Status

Documentation

Versioning

Adopters (base and end-user images)

Community discussion

Developing bootc

Governance

Badges

Code of Conduct

Attendees:

Agenda:

TODO:

Archive

Attendees:

Agenda:

TODO:

Archive

+ News +

Archive

A refresher on `tar`