Skip to content

on_boarding.md

Latest commit

 

History

History
143 lines (107 loc) · 16.1 KB

on_boarding.md

File metadata and controls

143 lines (107 loc) · 16.1 KB
graph LR
    CLI_Handler["CLI Handler"]
    Conductor["Conductor"]
    DAG_Manager["DAG Manager"]
    Database_Client["Database Client"]
    SQL_Dialect_Handler["SQL Dialect Handler"]
    Script_Handler["Script Handler"]
    Session_Manager["Session Manager"]
    CLI_Handler -- "initiates" --> Conductor
    Conductor -- "manages" --> DAG_Manager
    Conductor -- "uses" --> Database_Client
    Database_Client -- "uses" --> SQL_Dialect_Handler
    Conductor -- "manages" --> Session_Manager
    Session_Manager -- "processes" --> Script_Handler
    click CLI_Handler href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/CLI Handler.md" "Details"
    click Conductor href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/Conductor.md" "Details"
    click DAG_Manager href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/DAG Manager.md" "Details"
    click Database_Client href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/Database Client.md" "Details"
    click SQL_Dialect_Handler href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/SQL Dialect Handler.md" "Details"
    click Script_Handler href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/Script Handler.md" "Details"
    click Session_Manager href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/lea/Session Manager.md" "Details"
Loading

CodeBoardingDemoContact

Component Details

Lea is a data transformation tool that orchestrates the execution of SQL scripts within different database environments. The tool parses command-line arguments, constructs a directed acyclic graph (DAG) of SQL scripts based on their dependencies, and executes these scripts in the correct order using database-specific clients. It manages database sessions, handles SQL dialect variations, and provides mechanisms for testing data quality and managing audit tables. The core functionality revolves around transforming data through SQL scripts, ensuring data integrity, and supporting multiple database systems.

CLI Handler

The CLI Handler serves as the entry point for the Lea application. It parses command-line arguments using lea.cli:run and initiates the data transformation pipeline by calling the Conductor. It is responsible for setting up the environment and triggering the execution of the data transformation process based on user-provided arguments.

Related Classes/Methods:

Conductor

The Conductor orchestrates the entire data transformation process. It manages the DAG of scripts, prepares the database session, and executes the scripts in the correct order. It initializes database clients, manages audit tables, and handles the overall execution flow. The Conductor uses the DAG Manager to understand script dependencies and the Session Manager to execute scripts within a database session, ensuring that the data transformation is performed correctly and efficiently.

Related Classes/Methods:

DAG Manager

The DAG Manager constructs and manipulates the Directed Acyclic Graph (DAG) of SQL scripts. It determines the execution order of scripts based on their dependencies. It allows selecting subgraphs based on changed table references and iterating through ancestors and descendants of nodes in the graph. The DAG Manager provides the Conductor with the necessary information to execute scripts in the correct order, ensuring that dependencies are met before a script is executed.

Related Classes/Methods:

Database Client

The Database Client provides an abstraction layer for interacting with different database systems (BigQuery, DuckDB, MotherDuck, DuckLake). It handles tasks such as materializing scripts, querying data, cloning tables, deleting tables, and listing table statistics and fields. The Conductor uses the Database Client to execute scripts and manage data within a specific database, abstracting away the complexities of interacting with different database systems.

Related Classes/Methods:

SQL Dialect Handler

The SQL Dialect Handler provides dialect-specific functionality for parsing and formatting table references, converting table references to database-specific formats, and generating SQL code for data quality tests. The Database Client uses the SQL Dialect Handler to adapt SQL scripts to the specific database system being used, ensuring that the SQL scripts are compatible with the target database.

Related Classes/Methods:

Script Handler

The Script Handler is responsible for reading, parsing, and processing SQL scripts. It extracts dependencies, identifies assertion tests, and adds context to the scripts. The Conductor uses the Script Handler to understand the dependencies between scripts and to prepare them for execution, ensuring that all necessary information is available before a script is executed.

Related Classes/Methods:

Session Manager

The Session Manager handles the execution of SQL scripts within a database session. It adds context to scripts, runs scripts, monitors job execution, and promotes audit tables. The Conductor uses the Session Manager to manage the execution of scripts within a specific database session, ensuring that scripts are executed correctly and that audit tables are properly maintained.

Related Classes/Methods: