graph LR
Core_Linker_Orchestrator["Core Linker Orchestrator"]
Data_I_O_Backend_Abstraction["Data I/O & Backend Abstraction"]
Configuration_Setup["Configuration & Setup"]
Core_Linkage_Processing_Engine["Core Linkage Processing Engine"]
Reporting_Visualization["Reporting & Visualization"]
Core_Linker_Orchestrator -- "initiates data loading and manages data persistence via" --> Data_I_O_Backend_Abstraction
Core_Linker_Orchestrator -- "provides linkage settings to" --> Configuration_Setup
Core_Linker_Orchestrator -- "triggers the sequential stages within" --> Core_Linkage_Processing_Engine
Core_Linkage_Processing_Engine -- "requests data for processing and persists intermediate/final results through" --> Data_I_O_Backend_Abstraction
Core_Linkage_Processing_Engine -- "returns processed results to" --> Core_Linker_Orchestrator
Core_Linker_Orchestrator -- "requests reports and visualizations from" --> Reporting_Visualization
Reporting_Visualization -- "accesses raw and processed data from" --> Data_I_O_Backend_Abstraction
Reporting_Visualization -- "accesses settings from" --> Configuration_Setup
Reporting_Visualization -- "accesses model parameters/results from" --> Core_Linkage_Processing_Engine
click Core_Linker_Orchestrator href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/splink/Core_Linker_Orchestrator.md" "Details"
click Data_I_O_Backend_Abstraction href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/splink/Data_I_O_Backend_Abstraction.md" "Details"
click Configuration_Setup href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/splink/Configuration_Setup.md" "Details"
click Core_Linkage_Processing_Engine href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/splink/Core_Linkage_Processing_Engine.md" "Details"
click Reporting_Visualization href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/splink/Reporting_Visualization.md" "Details"
The splink project implements a robust data processing pipeline for record linkage, characterized by its modularity and pluggable backend architecture. At its core, the Core Linker Orchestrator directs the entire workflow, from initial data ingestion and configuration to the final output of linked records and analytical visualizations. Raw data is managed and accessed through the Data I/O & Backend Abstraction layer, which provides a unified interface to various data sources and database engines (e.g., DuckDB, Spark, PostgreSQL) by dynamically generating and executing SQL queries. User-defined linkage parameters are meticulously handled by the Configuration & Setup component, ensuring valid and consistent settings for the linkage process. The heart of the system is the Core Linkage Processing Engine, which encapsulates the sequential steps of blocking, comparison, statistical model training (EM), prediction, and clustering, transforming raw input into linked entities. Finally, the Reporting & Visualization component offers comprehensive tools for analyzing the linkage outcomes, model performance, and data characteristics, providing critical insights to users. This design emphasizes a clear, sequential data flow, with distinct component boundaries that facilitate both understanding and visual representation in a flow graph diagram.
Core Linker Orchestrator [Expand]
The central control unit managing the entire record linkage workflow, coordinating the execution and data flow between all other components.
Related Classes/Methods:
Data I/O & Backend Abstraction [Expand]
Manages data loading from various sources (e.g., Pandas, Spark DataFrames, database tables) and abstracts interactions with different database backends (DuckDB, Spark, Postgres, etc.) through SQL generation and execution. It serves as the pluggable data access layer.
Related Classes/Methods:
splink.internals.database_apisplink.internals.splink_dataframesplink.internals.athena.database_apisplink.internals.duckdb.database_apisplink.internals.postgres.database_apisplink.internals.spark.database_apisplink.internals.sqlite.database_apisplink.internals.pipelinesplink.internals.dialectssplink.internals.sql_transformsplink.internals.term_frequenciessplink.internals.vertically_concatenate
Configuration & Setup [Expand]
Handles the definition, parsing, and validation of all linkage settings, including blocking rules, comparison levels, and model parameters, ensuring consistency and validity before pipeline execution.
Related Classes/Methods:
splink.internals.settingssplink.internals.settings_creatorsplink.internals.validate_jsonschemasplink.internals.settings_validation
Core Linkage Processing Engine [Expand]
Executes the main record linkage pipeline steps: generating candidate pairs (blocking), comparing attributes to create comparison vectors, training the statistical model (EM algorithm), predicting match probabilities, and clustering linked records.
Related Classes/Methods:
splink.internals.blockingsplink.internals.blocking_rule_creatorsplink.internals.blocking_rule_librarysplink.internals.blocking_analysissplink.internals.comparisonsplink.internals.comparison_creatorsplink.internals.comparison_levelsplink.internals.comparison_level_librarysplink.internals.column_expressionsplink.internals.input_columnsplink.internals.expectation_maximisationsplink.internals.em_training_sessionsplink.internals.estimate_usplink.internals.m_u_records_to_parameterssplink.internals.predictsplink.internals.miscsplink.internals.clusteringsplink.internals.one_to_one_clusteringsplink.internals.connected_componentssplink.internals.edge_metrics
Reporting & Visualization [Expand]
Provides tools for generating interactive charts and reports to analyze the linkage process, model performance, and results, offering insights into data quality and linkage outcomes.
Related Classes/Methods: