Caddy 2.11.1: socket activation problems with ACME

[TNorthover]

Issue Details

Since updating to 2.11, the ACME path seems to have problems with socket activation. The (slightly sanitized) error log is:

Feb 25 06:36:14 caddy caddy[1037]: {"level":"info","ts":1772001374.5532987,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"XYZ.mydomain.com"}
Feb 25 06:36:14 caddy caddy[1037]: {"level":"info","ts":1772001374.5535624,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["XYZ.mydomain.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"email@address"}
Feb 25 06:36:14 caddy caddy[1037]: {"level":"info","ts":1772001374.5535676,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["XYZ.mydomain.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"email@address"}
Feb 25 06:36:15 caddy caddy[1037]: {"level":"info","ts":1772001375.363842,"msg":"trying to solve challenge","identifier":"XYZ.mydomain.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Feb 25 06:36:15 caddy caddy[1037]: {"level":"error","ts":1772001375.644645,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"XYZ.mydomain.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[XYZ.mydomain.com] solving challenges: presenting for challenge: presenting with embedded solver: could not start listener for challenge server at fd/4:80: listen tcp: lookup fd/4: no such host (order=https://acme-v02.api.letsencrypt.org/acme/order/SENSITIVE) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

which suggests the Caddyfile bind argument of fd/4 is making its way to ACME library and being fed to an important DNS query.

A reasonably minimal Caddyfile I can use to reproduce the issue is:

{
        auto_https disable_redirects
	default_bind fd/4 {
                protocols h1 h2
        }
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

http://
        bind fd/3 {
                protocols h1
        }
        redir https://{host}{uri} 308
}

wibble.mydomain.com {
	respond "Hello"
}

In this scheme I'm using systemd+Podman to feed port 80 in as fd/3 and port 443 as fd/4.

Assistance Disclosure

No response

If AI was used, describe the extent to which it was used.

No response

[francislavoie]

We reverted the socket activation changes merged since 2.10.2 if that's what you're relying on. We had arguments about the correct approach to take, and we did not have time to resolve it in time for the release.

[TNorthover]

I upgraded my images directly from 2.10.2 (which Works for Me(TM)) to 2.11.1 so I don't think I'd have been affected by any churn that went on during development there.

[francislavoie]

I'm not seeing any changes between 2.10.2 and 2.11.1 relating to socket activation or fd.

[TNorthover]

I didn't see anything suspicious in the ACME handling either, so perhaps an updated library dependency is doing something new with input it already had?

I've rebuilt 2.10.2 and 2.11.2 locally and both exhibit the behaviour described, so it at least looks like it's something pinned rather than a floating dep.

[Monviech]

I have a feeling its this:

#7278

I would not blindly revert it though as it solved a real other bug. I have no experience with socket activation, maybe this needs another fix to handle such a socket correctly.

Previously it worked by accident most likely since default_bind just bound to the wildcard interface and not some file descriptor.

EDIT: The ACME client most likely needs to become socket activation aware like the HTTP app (I didnt check this assumption). That it fails now should prove this was a prior undiscovered edge case.

[francislavoie]

Ah, that makes sense. In that case @TNorthover please try removing default_bind and use bind in each site for now, does that work?

[TNorthover]

Yep, that seems to do the trick. Thanks for the help.

[Monviech]

@francislavoie Do you want input validation or is the edge case okay for now? Nothing seems to fail hard, the ACME client just treats it like a hostname.

[francislavoie]

We should port the fd stuff to certmagic I think. But fd stuff is currently up in the air because of arguments about it.

[passcod]

I'm affected by this as well, all my sites are breaking on upgrade to caddy 2.11.1. To be clear, are you saying that caddy has removed fd support?

[m0canu1]

I'm affected by this as well, all my sites are breaking on upgrade to caddy 2.11.1. To be clear, are you saying that caddy has removed fd support?

No, from what I understood the issue is that certmagic (the one that handles certificate renewal) doesn't recognize fd and tries to bind port :80 without success.

[TNorthover]

To be clear, are you saying that caddy has removed fd support?

It's still there, but for now you need to replace the single default_bind directive you have with a separate bind for each https://... { } handler. I put mine in a snippet and imported it each time to save a bit of duplication.

[m0canu1]

Can you share your snippet? I created the following but keep getting bind error, this time on :443

# Define once, import everywhere
(tls_binds) {
    bind fd/4 {
        protocols h1 h2
    }
    bind fdgram/5 {
        protocols h3
    }
}

sub.tld.com {
    import tls_binds
    import web_defaults
    import custom_log sub.tld.com
    reverse_proxy http://192.168.1.1:8080
}

this is the error:

{"level":"error","ts":1772790657.55009,"logger":"tls.renew","msg":"will retry","error":"[sub.tld.com] Renew: [sub.tld.com] solving challenges: presenting for challenge: presenting with embedded solver: could not start listener for challenge server at :443: listen tcp :443: bind: permission denied (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/195643044/33867204413) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":63.369863479,"max_duration":2592000}

[TNorthover]

That's pretty much exactly what I've got. Are you maybe trying to deploy it piecewise?

If some subdomains aren't yet using the new tls_binds then they'll still try to grab the real port 443 and won't be allowed, which would produce an error very much like you're seeing.

[m0canu1]

I added the snippet tls_binds to all SNIs of my Caddyfile, if this is your question.