Skip to content

enhancement(kafka source): add multithreading option for kafka source #24550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
esensar wants to merge 28 commits into
base: master
Choose a base branch
from
+403 −96
Open

enhancement(kafka source): add multithreading option for kafka source #24550

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8862b4c
enhancement(kafka source): add multithreading option for kafka source
esensar Jan 27, 2026
43bd002
Add changelog entry
esensar Jan 27, 2026
23502e1
Make active tasks num naming more consistent
esensar Jan 27, 2026
62b550b
Add a custom `OwnedMessage` type to reduce payload copying
esensar Jan 27, 2026
78c9b92
Merge branch 'master' into enhancement/kafka-source-multithreading
esensar Jan 27, 2026
0c00ec8
Default to available cores for `max_message_handling_tasks`
esensar Jan 28, 2026
c9480dc
Handle both errors and oks from batch
esensar Mar 18, 2026
45e2681
Merge branch 'master' into enhancement/kafka-source-multithreading
esensar Mar 18, 2026
d42eebd
Fix max_message_handling_tasks check to include equality
esensar Mar 18, 2026
b57605b
Use `fetch_update` for atomic active_tasks update
esensar Mar 20, 2026
522f0e8
Merge branch 'master' into enhancement/kafka-source-multithreading
esensar Mar 20, 2026
548355f
Update LICENSE-3rdparty
esensar Mar 20, 2026
d7e5b29
Update licenses
thomasqueirozb Mar 20, 2026
671aef1
Use semaphore instead of counting active tasks
esensar Mar 26, 2026
bcca2c5
Update `max_message_handling_tasks` docs
esensar Mar 26, 2026
1e925f5
Use `NonZeroUsize` for `max_message_handling_tasks`
esensar Mar 27, 2026
93f3ad8
Add a channel for done futures to prevent polling empty FuturesOrdered
esensar Apr 15, 2026
1ca7893
Prevent blocking on notifying that task is done
esensar Apr 16, 2026
fed65b0
Merge branch 'master' into enhancement/kafka-source-multithreading
esensar Apr 17, 2026
098650d
Fix bias issue with future done rx
esensar Apr 17, 2026
086646b
Merge branch 'master' into enhancement/kafka-source-multithreading
pront May 19, 2026
6194fe7
Delay Eof handling when multithreading is enabled
esensar May 20, 2026
1daa4c5
Fix streams ordering issue by using iter instead of select_all
esensar May 20, 2026
ce071b1
Add multithreaded variants to integration tests
esensar May 21, 2026
5658c3e
Log joinerrors
esensar May 21, 2026
12a501e
Cancel all pending messages processing when finalizer is dropped
esensar May 21, 2026
084cbe9
Handle invalid number of max_message_handling_tasks
esensar May 21, 2026
6135c05
Ensure order of events is kept in multithreaded mode
esensar May 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog.d/24550_kafka_source_multithreading.enhancement.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Adds `multithreading` configuration option to `kafka` source, enabling increased throughput.

authors: esensar Quad9DNS
Loading
Loading