Coordinated support for immutable distributions

[martinjungblut]

Disclaimer

I'm the PCP package maintainer at SUSE, and one of the main
contributors to openSUSE's PCP package.

Background

A growing number of distributions ship with a read-only /usr and a
writable /var on a separate subvolume, with transactional updates
that snapshot /usr independently of /var.

Examples include Fedora Atomic and openSUSE MicroOS.

On these distributions, files that the PCP package ships under
/var/lib/pcp/ are problematic:

• They live on a subvolume not covered by /usr snapshots, so they
  don't roll back with the rest of the system.
• They're flagged by distro packaging checks (e.g. rpmlint's
  dir-or-file-outside-snapshot warning on openSUSE, and the broader
  rpm-ostree expectations on Fedora variants).
• Some downstream distros refuse to publish builds that violate this.

This has been discussed in this tracker before, partially, through issues #816 and #1418.

To work around this, distributions are converging on a standard
pattern: symlinking /var directories to read-only locations in
/usr or /etc.
SUSE is about to ship an implementation using systemd-tmpfiles.
This involves recreating the /var/lib/pcp/ tree at boot with
symlinks targeting /usr/libexec/pcp/ and /etc/pcp/, while
relocating the testsuite's files to /usr/libexec/pcp/stash/.

The intent behind this issue is to ask whether upstream would welcome
a more coordinated approach, so each distro isn't re-inventing the
wheel slightly differently.

What we built downstream (for context)

Briefly, the SUSE approach is:

1. A buildroot-relocation helper runs after make install_pcp.
   For each subpackage's file list, it classifies entries under
   /var/lib/pcp/ by type (symlink / regular file / directory).
2. Pre-existing symlinks (most PMDA content) get %ghost'd in the RPM
   and an L+ tmpfiles.d entry is emitted pointing at the original
   /usr/libexec/pcp/... or /etc/pcp/... target.
3. Real files (only pcp-testsuite) are cp -a'd to
   /usr/libexec/pcp/stash/, %ghost'd at the legacy location, and
   an L+ entry points at the stash copy.
4. systemd-tmpfiles recreates the legacy /var/lib/pcp/ tree at boot,
   so PCP's runtime sees the layout it was built for.

The runtime environment is unchanged: PCP_VAR_DIR=/var/lib/pcp stays
correct, no PCP source is patched, regular non-transactional installs
are unaffected.

We can share the spec and helper script if useful. We're not proposing
them as-is for upstream adoption, as they're SUSE-specific, but they
may be useful as a reference for what works.

I haven't checked what Fedora is doing in detail, although that may be
interesting as well.

Open questions & next steps

We'd like to open a discussion on whether upstream is interested in coordinating support for immutable distributions, or if this should remain a per-distro effort indefinitely.

If upstream consolidation makes sense, potential paths include:

1. Documenting a standard symlink pattern.
2. Shipping an upstream build helper to generate tmpfiles.d snippets.
3. Adding a /usr-only configure flag.
4. Reconsidering the file layout to move static content out of /var.
   Note: This was ruled out for the SUSE-specific packages as simply changing PCP_VAR_DIR might break established documentation and runtime expectations.

We are happy to share our SUSE implementation, help prototype, and test patches.
Even a brief signal on your preferred direction, or confirmation that
we should continue handling this downstream, would be incredibly
helpful for us to plan our next steps.

[natoscott]

@martinjungblut hi! @ddiss was our previous SuSE maintainer contact, really nice to meet more SuSE contributors. 💚

The intent behind this issue is to ask whether upstream would welcome
a more coordinated approach, so each distro isn't re-inventing the
wheel slightly differently.

Sounds good. Fedora and RHEL have shipped many of the things you talk about here - so I'd first do a detailed audit of what exists in upstream PCP today (Fedora and RHEL do not tend to have any patches that are not upstream, and certainly not in this area), where this is considered a solved problem.

From memory the initial /var symlinking and /etc transition work started back in 2020 via commit faeb250 and since then full transition to tmpfiles.d was also completed, incl. with the pcp user/group account setup that way now.

This transition happened over a few years and its all quite stable these days. If there are specific areas that are causing problems on the SuSE distros, I think it'd be best to tackle each through small, focused GH issues/PRs so any impact on the Fedora/RHEL equivalent concepts can be assessed and discussed. The spec file used as the base for both Fedora and RHEL is build/rpm/redhat.spec - definitely worth comparing this to current SuSE spec files (I haven't done that though).

We have tended to consider the testsuite as not worth updating, as it works for us as-is and rollback of testsuite results etc is not useful to anyone (nowadays the testsuite is only made available internally for RHEL FWIW).

[natoscott]

@martinjungblut I'll close this for now with lack of response. Red Hat rpm packages are working well in our immutable distros but please do propose any additional build changes you require for SuSE distros (each could be a new PR, no GH issues needed - they'll be accepted provided nothing regresses for others).