Skip to content

Stream_Processing_Engine.md

Latest commit

 

History

History
74 lines (43 loc) · 3.66 KB

Stream_Processing_Engine.md

File metadata and controls

74 lines (43 loc) · 3.66 KB
graph LR
    StreamingDataFrame["StreamingDataFrame"]
    Stream["Stream"]
    StreamingSeries["StreamingSeries"]
    Window["Window"]
    Join["Join"]
    StreamingDataFrame -- "defines processing logic for" --> Stream
    StreamingDataFrame -- "configures" --> StreamingSeries
    StreamingDataFrame -- "configures" --> Window
    StreamingDataFrame -- "configures" --> Join
    Stream -- "receives definitions from" --> StreamingDataFrame
    Stream -- "orchestrates" --> StreamingSeries
    Stream -- "orchestrates" --> Window
    Stream -- "orchestrates" --> Join
    StreamingSeries -- "applies transformations to data flow via" --> Stream
    Window -- "processes results with" --> Stream
    Join -- "integrates data into pipeline with" --> Stream
Loading

CodeBoardingDemoContact

Details

The Stream Processing Engine subsystem is the core of the quix-streams project, providing the declarative API and underlying mechanisms for real-time data transformations. It consumes messages from Kafka, processes them according to defined logic, and produces results.

StreamingDataFrame

Provides the high-level, declarative API for users to define data transformations, aggregations, and routing logic on streaming data. It acts as the primary interface for building stream processing pipelines.

Related Classes/Methods:

Stream

The core internal component that manages the actual sequence and lineage of operations applied to a data stream. It represents the underlying processing graph and executes the transformations defined by StreamingDataFrame.

Related Classes/Methods:

StreamingSeries

Enables various element-wise and column-wise operations (arithmetic, comparison, logical, data manipulation) on individual data series (columns) within a StreamingDataFrame. It provides the granular data manipulation capabilities.

Related Classes/Methods:

Window

Defines and manages windows of data for aggregations (e.g., tumbling, hopping, sliding windows). It handles the logic for collecting, aggregating, and expiring data within these windows, crucial for time-based stream processing.

Related Classes/Methods:

Join

Provides mechanisms to join streaming data with other streams or external lookup tables. It manages data fetching, caching, and processing of join keys, enabling complex data correlation across streams.

Related Classes/Methods:

FAQ