Overlayed ports not reflected in ports tree config

[michael-o]

Prerequisites

• Have you checked for an existing issue describing your problem?
• Are you running the latest version?
• Is your ports tree recent?
• Is your FreeBSD Host on a supported release?

Describe the bug

I wanted to try the new overlay feature w/o resorting to portshaker. While it works the ports tree label does not contain the overlayed repos.

How to reproduce

My command was:

poudriere bulk -j 122-release-amd64 -p default -O ldadw   -f /usr/local/etc/poudriere.d/122-release-amd64-ldadw-ports

and the build was called: 122-release-amd64-default

Expected behavior

I would have expected that the build woudl be called 122-release-amd64-default,overlay. All repos concatenated together. With the missing information it makes it really hard to provide make.conf and options for this specific overlay.

Environment

• Host OS [e.g. 12.2 amd64]: 12.2-STABLE on amd64
• Jail OS [e.g. 12.0 powerpc]: 12.2-RELEASE on amd64
• Poudriere Version [e.g. 3.3.1 or git hash or port version]: 93ceb26
• Ports branch and revision [e.g. 2020Q3 r550754]: main

Additional context

My custom ports tree ldadw contains only company-internal ports which aren't in the default ports tree, so they are disjoint.

[bdrewery]

I kind of agree with you but adding -O into the mix for the config files and dirs is a whole bunch of complexity. The workaround here is simply adding -z ldadw as well.

[michael-o]

I kind of agree with you but adding -O into the mix for the config files and dirs is a whole bunch of complexity. The workaround here is simply adding -z ldadw as well.

That's what I would do actually, but the concatenated tree would just replace the single ports tree. I would not expect you to provde the following: 122-release-amd64-<tree>-<overlays>-<set>..., but overlays to be just in <tree>. This should reduce complexity.
Alternatively one can configure ports overlays as alias with a command and use that alias and poudriere would do that the internally. WDYT?

[michael-o]

This block hasn't been updated:

[bdrewery]

After using overlays myself for a while I'm more confused on the right path forward. There's 0 indication anywhere in the log files or web about the use of overlays. And there's no way to customize options, make.conf, or poudriere.conf, when using specific overlays. It really smells like a superset of -z.

[michael-o]

This is my understanding of port overlays: POs are not intended to be complete (necessarily), they override or extend the FreeBSD ports tree; they cannot exist w/o the ports tree. The result will be a (super set, aggregated) ports tree. The same way I would treat all config files: -P default -O foo -O bar -O baz builds the tree default+foo+bar+baz (plus (+) is a sample separator), therefore first all options, make.conf, poudriere.conf of default are copied, then the same files from foo, bar, and baz override sequentially previously set/copied/merged files unless you have config files which are explicitly for this tree combination. (based on https://github.com/freebsd/poudriere/wiki/poudriere.8#custom--build-options).
Now if you use sets, they come on top, so alternations for this previous aggregation.

WDYT?

[bdrewery]

Yeah I think what you're saying is to add the overlays into the computed build "mastername".
However, thinking about it more it really does seem like combinations of -z, -p, and -j should already be covering this. Those, as you point out in your link, already read in dynamic configs / options. If there is actual demand for more than the analogous 3 -O this provides then we can explore supporting multiple -z at that point.
So with that mindset the problem I see is that -O exists. I am thinking of only allowing adding overlays via *poudriere.conf.

Here's a mockup idea that avoids using OVERLAYS=whatever because since a simple append like OVERLAYS+= won't work which I think makes setting like that annoying for users.

foo-poudriere.conf

overlay foo

poudriere.conf

overlay all

So using -z foo would get overlays, in this order, foo all. Using no -z would get all.

[michael-o]

Yeah I think what you're saying is to add the overlays into the computed build "mastername".
Exactly!

I don't fully understand you proposal yet, but wouldn't that mean that I cannot provide say options for one overlay or any other configuration? I see overlays only as a layer on top of -p never standalone.

I still think that a computed build mastername thus the configuration for it is more accessible that forcing people to use sets when they don't need it. I see sets as alternations of the former.

[bdrewery]

Questions to help me understand your wants here.

What use case is there for an overlay where not using a set?

What does set add on top of an overlay that an overlay wouldn't want?

Why would you want to use the same set with different overlays?

[michael-o]

Let me think about it over the weekend and I will provide you answers. Thank you!

[grahamperrin]

👀