chore(deps): link rdkafka dynamically in dev builds#25439
Draft
pront wants to merge 7 commits into
Draft
Conversation
Drops `rdkafka?/gssapi` from the `default` Cargo feature so dev builds (`cargo build`, `make build`) compile librdkafka without GSSAPI/Kerberos SASL support. As a direct consequence, `libsasl2-dev` / `cyrus-sasl` no longer needs to be installed for local development. Splits the previous `vendored` umbrella into three composable features: - `gssapi`: dynamic linkage against system libsasl2. Reuses pre-PR `default` semantics on macOS (where Apple ships libsasl2 + GSS). - `gssapi-vendored`: static-bundled libsasl2 + krb5-src. Linux-only in practice; the bundled path does not link on macOS arm64 (unresolved gss_* symbols). - `vendored`: umbrella for static C deps, currently just `gssapi-vendored`. Intended to grow as more vendored deps land. Adds a `target-aarch64-apple-darwin` entry to the target-* feature table so the apple-darwin publish job picks up `gssapi` via Cargo rather than overriding `FEATURES` inline. Released macOS binaries retain GSSAPI via the same dynamic linkage as pre-PR. Linux x86_64-gnu archives are byte-equivalent (same vendored static linkage as before). Updates `docs/DEVELOPING.md` to document the three options and their prereqs, and adds `gssapi` to the spelling allow-list. Verified locally on aarch64-apple-darwin: `cargo check --release --no-default-features --features default,gssapi`. Follow-up: flip `default` to `rdkafka?/dynamic_linking` so dev builds link to a system librdkafka and skip the multi-minute source compile. An earlier attempt on this branch was reverted because Ubuntu 24.04's apt librdkafka-dev ships 2.3.0 but rdkafka-sys 4.10.0+2.12.1 requires 2.12.1+; the follow-up will install/cache a compatible librdkafka in the setup action. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Builds librdkafka from source into a per-version prefix under $HOME so CI (via actions/cache) and local developers on distros with older packages can land on a version compatible with rdkafka-sys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `librdkafka` input to .github/actions/setup that caches and installs librdkafka 2.12.1 (built via install-librdkafka.sh) and exposes it through PKG_CONFIG_PATH + LD_LIBRARY_PATH. Opt-in for workflows that build Vector with the `default` feature (clippy, generated-docs, msrv, unit-tests, test-make-command, and component-features). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enables `rdkafka?/dynamic_linking` in the `default` Cargo feature so dev builds (`cargo build`, `make build`) link against a system librdkafka via pkg-config instead of compiling rdkafka-sys' bundled C source on every clean build. Release feature sets (`default-cmake`, `default-musl`, `default-msvc`, `target-*`) don't inherit `default` and still build librdkafka from the vendored source via `cmake_build`, so release archives are unchanged. Updates docs/DEVELOPING.md with the local prerequisite (librdkafka >= 2.12.1) and points Linux developers at install-librdkafka.sh since Ubuntu's apt librdkafka-dev is too old. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
domain: ci
The actions/cache wiring can land as a follow-up once the dynamic linking is shipping in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ```bash | ||
| curl -fsSL https://packages.confluent.io/clients/deb/archive.key \ | ||
| | sudo gpg --dearmor -o /usr/share/keyrings/confluent-archive-keyring.gpg | ||
| echo "deb [signed-by=/usr/share/keyrings/confluent-archive-keyring.gpg] https://packages.confluent.io/clients/deb $(lsb_release -cs) main" \ |
Replace the from-source build with a Confluent apt install. Faster on CI (seconds vs minutes), removes the need for actions/cache, and drops the install-librdkafka.sh helper. Local Linux devs follow the same recipe; documented in docs/DEVELOPING.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pront
force-pushed
the
chore/kafka-dev-build-dynamic-linking
branch
from
f66a307 to
25063b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on top of #25256.
Flips the
defaultCargo feature to enablerdkafka?/dynamic_linkingso dev builds (cargo build,make build) link against a system librdkafka via pkg-config instead of compiling rdkafka-sys' bundled C source on every clean build. Release feature sets (default-cmake,default-musl,default-msvc,target-*) do not inheritdefaultand continue to build librdkafka from the vendored source viacmake_build, so release archives are unchanged.The blocker for the previous attempt was that Ubuntu 24.04's distro
librdkafka-devships 2.3.0 whilerdkafka-sys 4.10.0+2.12.1requires>= 2.12.1. This PR addresses that by installing librdkafka from Confluent's apt repo (upstream maintainers; ships 2.10.x through 2.14.x for Ubuntu noble), wired through a newlibrdkafkainput on.github/actions/setup. Opt-in for the dev-build workflows (clippy, generated-docs, msrv, unit-tests, test-make-command, component-features).Local-dev impact:
brew install librdkafka.docs/DEVELOPING.md.Vector configuration
N/A — build-system change only.
How did you test this PR?
--features default,sources-kafka. Cargo's JSON build artifact output confirmslinked_paths=["native=/opt/homebrew/Cellar/librdkafka/2.13.2/lib"]andlinked_libs=["rdkafka"], i.e. dynamic linking is active and no bundled compile happens.target-x86_64-unknown-linux-gnu,target-aarch64-apple-darwin) are unaffected: cargo tree still routes them throughcmake_build/vendoredas before.packages.confluent.io/clients/deb/dists/nobleships librdkafka-dev 2.12.1, 2.12.0, 2.13.0/2, 2.14.0/1.Change Type
Is this a breaking change?
Release archives and runtime behavior are unchanged. Dev/CI builds change their linkage strategy for librdkafka only.
Does this PR include user facing changes?
no-changeloglabel to this PR.Developer-facing only: Cargo feature wiring and dev/CI build prerequisites.