graph LR
Sink["Sink"]
SinkManager["SinkManager"]
Batch["Batch"]
DataFrameSinkIntegration["DataFrameSinkIntegration"]
ElasticsearchSink["ElasticsearchSink"]
InfluxDB3Sink["InfluxDB3Sink"]
FileDestination["FileDestination"]
DataFrameSinkIntegration -- "directs data to" --> SinkManager
SinkManager -- "orchestrates" --> Sink
ElasticsearchSink -- "implements" --> Sink
InfluxDB3Sink -- "implements" --> Sink
FileDestination -- "implements" --> Sink
ElasticsearchSink -- "utilizes" --> Batch
InfluxDB3Sink -- "utilizes" --> Batch
FileDestination -- "utilizes" --> Batch
The Data Sinks subsystem is primarily defined by the quixstreams/sinks/ directory. It encompasses all modules responsible for publishing processed data from Kafka topics to various external systems, acting as the egress points for data from the streaming application.
The foundational abstract class (quixstreams.sinks.base.sink.Sink) that defines the contract for all data sinks. It ensures a consistent API for operations such as setup, writing data, flushing buffers, and starting/stopping the sink. This aligns with the "Modular and Extensible" architectural bias, allowing new sinks to be easily integrated.
Related Classes/Methods:
Responsible for managing the lifecycle and orchestration of multiple Sink instances within the application (quixstreams.sinks.base.manager.SinkManager). It handles the registration, initialization, and coordination of all active sinks, ensuring they operate correctly as part of the overall data pipeline. This component embodies the "Application Orchestrator" aspect for the sink side.
Related Classes/Methods:
A utility component (quixstreams.sinks.base.batch.Batch) that collects and batches SinkItem objects. Its primary responsibility is to optimize write operations to external systems by reducing the number of individual write calls, thereby improving efficiency and throughput. This supports the "Reliability and Scalability" emphasis.
Related Classes/Methods:
Serves as the bridge between the StreamingDataFrame processing logic and the Data Sinks subsystem (quixstreams.dataframe.dataframe.sink). It's the point where processed data from the DataFrame is directed towards the configured sinks.
Related Classes/Methods:
A concrete implementation of the Sink interface for Elasticsearch (quixstreams.sinks.community.elasticsearch.ElasticsearchSink). It encapsulates the logic for connecting, authenticating, formatting data, and writing to Elasticsearch.
Related Classes/Methods:
A concrete implementation of the Sink interface for InfluxDB3 (quixstreams.sinks.core.influxdb3.InfluxDB3Sink). It encapsulates the logic for connecting, authenticating, formatting data, and writing to InfluxDB3.
Related Classes/Methods:
A base class for file-based sink implementations (quixstreams.sinks.community.file.destinations.base.FileDestination). It encapsulates the common logic for connecting, authenticating, formatting data, and writing to various file systems.
Related Classes/Methods: