graph LR
Job_Orchestration_Workflow_Definition["Job Orchestration & Workflow Definition"]
Workflow_Execution_Data_Management["Workflow Execution & Data Management"]
Runner_Execution_Environment["Runner Execution Environment"]
Quality_Control_Termination["Quality Control & Termination"]
Common_Utilities_Notifications["Common Utilities & Notifications"]
Job_Orchestration_Workflow_Definition -- "initiates job processing" --> Workflow_Execution_Data_Management
Job_Orchestration_Workflow_Definition -- "defines execution patterns for" --> Workflow_Execution_Data_Management
Job_Orchestration_Workflow_Definition -- "utilizes" --> Common_Utilities_Notifications
Job_Orchestration_Workflow_Definition -- "leverages" --> Common_Utilities_Notifications
Job_Orchestration_Workflow_Definition -- "validates with" --> Quality_Control_Termination
Workflow_Execution_Data_Management -- "executes commands via" --> Runner_Execution_Environment
Workflow_Execution_Data_Management -- "interacts with" --> Quality_Control_Termination
Workflow_Execution_Data_Management -- "applies transformations using" --> Common_Utilities_Notifications
Runner_Execution_Environment -- "reports status to" --> Workflow_Execution_Data_Management
click Job_Orchestration_Workflow_Definition href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/BayerCLAW/Job Orchestration & Workflow Definition.md" "Details"
click Workflow_Execution_Data_Management href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/BayerCLAW/Workflow Execution & Data Management.md" "Details"
click Runner_Execution_Environment href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/BayerCLAW/Runner Execution Environment.md" "Details"
click Quality_Control_Termination href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/BayerCLAW/Quality Control & Termination.md" "Details"
click Common_Utilities_Notifications href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/BayerCLAW/Common Utilities & Notifications.md" "Details"
This graph illustrates the architecture of BayerCLAW, a system designed for orchestrating and executing complex bioinformatics workflows. The main flow involves the Job Orchestration & Workflow Definition component, which handles initial job setup, routing, and compiles high-level workflow definitions into executable state machine language. This compiled workflow is then passed to the Workflow Execution & Data Management component, which manages the actual execution, data handling, and specific workflow patterns like scatter-gather. The Runner Execution Environment provides the isolated environment for command execution. Throughout the process, Quality Control & Termination ensures correctness and manages instance lifecycle, while Common Utilities & Notifications provides shared services for data manipulation and communication.
Manages the initial setup, routing, and comprehensive compilation of high-level workflow definitions into executable AWS Step Functions state machine language, including handling various step types (batch, scatter-gather, parallel, sub-pipeline, native, chooser) and their validation.
Related Classes/Methods:
BayerCLAW.lambda.src.initializer.initializer(full file reference)BayerCLAW.lambda.src.router.job_router(full file reference)BayerCLAW.lambda.src.job_def.register(full file reference)BayerCLAW.lambda.src.compiler.pkg.compiler(full file reference)BayerCLAW.lambda.src.compiler.pkg.state_machine_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.util(full file reference)BayerCLAW.lambda.src.compiler.pkg.validation(full file reference)BayerCLAW.lambda.src.compiler.pkg.scatter_gather_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.chooser_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.enhanced_parallel_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.subpipe_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.native_step_resources(full file reference)BayerCLAW.lambda.src.compiler.pkg.batch_resources(full file reference)BayerCLAW.lambda.src.chooser.multichooser(full file reference)
The central control component for the bclaw_runner, managing the entire job execution lifecycle, including data handling (S3 repository interactions, caching), and facilitating data exchange for sub-pipelines and scatter-gather operations during execution.
Related Classes/Methods:
BayerCLAW.bclaw_runner.src.runner.runner_main(full file reference)BayerCLAW.bclaw_runner.src.runner.preamble.log_preamble(7:14)BayerCLAW.bclaw_runner.src.runner.tagging.tag_this_instance(12:27)BayerCLAW.bclaw_runner.src.runner.repo(full file reference)BayerCLAW.bclaw_runner.src.runner.cache(full file reference)BayerCLAW.lambda.src.subpipes.subpipes(full file reference)BayerCLAW.lambda.src.scatter.scatter(full file reference)
Manages the local execution environment for the runner, responsible for running user-defined commands within that workspace and handling the execution of child containers using Docker-in-Docker.
Related Classes/Methods:
BayerCLAW.bclaw_runner.src.runner.workspace(21:35)BayerCLAW.bclaw_runner.src.runner.dind(full file reference)BayerCLAW.bclaw_runner.src.runner.signal_trapper.signal_trapper(26:42)
Performs quality control checks both within the AWS Lambda environment and during job execution, and manages the graceful termination of instances, particularly for spot instances.
Related Classes/Methods:
BayerCLAW.lambda.src.qc_checker.qc_checker(full file reference)BayerCLAW.bclaw_runner.src.runner.qc_check(full file reference)BayerCLAW.bclaw_runner.src.runner.termination(full file reference)
Provides shared utility functions for data manipulation, including general substitutions, repository operations, reading various file formats, and generating/sending notifications related to workflow state changes.
Related Classes/Methods:
BayerCLAW.lambda.src.common.python.substitutions(full file reference)BayerCLAW.lambda.src.common.python.repo_utils(full file reference)BayerCLAW.lambda.src.common.python.file_select(full file reference)BayerCLAW.lambda.src.notifications.notifications(full file reference)BayerCLAW.bclaw_runner.src.runner.string_subs(full file reference)