graph LR
redun_value["redun.value"]
redun_file["redun.file"]
redun_hashing["redun.hashing"]
redun_backends_base["redun.backends.base"]
redun_backends_db["redun.backends.db"]
redun_backends_value_store["redun.backends.value_store"]
redun_tags["redun.tags"]
redun_handle["redun.handle"]
redun_value -- "uses" --> redun_hashing
redun_file -- "extends" --> redun_value
redun_file -- "inherits from" --> redun_value
redun_file -- "uses" --> redun_hashing
redun_backends_db -- "implements" --> redun_backends_base
redun_backends_base -- "interacts with" --> redun_value
redun_backends_db -- "relies on" --> redun_value
redun_backends_value_store -- "leverages" --> redun_file
redun_backends_db -- "utilizes" --> redun_backends_value_store
redun_backends_db -- "manages" --> redun_tags
redun_handle -- "inherits from" --> redun_value
redun_backends_db -- "manages" --> redun_handle
This component is fundamental to Redun's reproducibility, caching mechanisms, and historical analysis capabilities by managing the persistence, serialization, hashing, and retrieval of all workflow metadata, task values, and files.
This is the foundational module defining the Value base class, from which nearly all Redun data types (including expressions, tasks, files, and handles) inherit. It provides core mechanisms for consistent serialization, deserialization, and hashing of Python objects. The TypeRegistry within this module is crucial for managing how different data types are handled for persistence and caching.
Related Classes/Methods:
This module extends the redun.value abstraction to specifically handle file and directory objects. It provides a unified interface for interacting with various file systems (e.g., local, S3, Azure Blob, GCP Storage, HTTP, FTP), abstracting away the underlying storage details. It ensures that files are treated as first-class Redun values, allowing them to be hashed, cached, and tracked within workflows.
Related Classes/Methods:
This module provides the core logic for consistent hashing of Python objects and data structures. It is a cornerstone of Redun's caching mechanisms and reproducibility, ensuring that identical inputs always produce identical hashes, allowing for efficient cache lookups and verification of computational integrity.
Related Classes/Methods:
This module defines the abstract base classes and interfaces for Redun backends. It establishes the contract that any concrete backend implementation must adhere to for persisting and retrieving workflow metadata, job status, and cached results. It ensures a consistent API for different storage solutions.
Related Classes/Methods:
This is a concrete implementation of redun.backends.base, providing a robust database-backed persistence layer for Redun. It handles the storage and retrieval of detailed workflow metadata, including executions, jobs, call nodes, values, and tags. It leverages internal serializers (redun.backends.db.serializers) to manage the structured storage of these entities.
Related Classes/Methods:
redun.backends.db(1:1)
This component, often used in conjunction with redun.backends.db, is responsible for the efficient storage and retrieval of the actual content of large values (e.g., task outputs, intermediate data). It likely leverages file system abstractions provided by redun.file for its operations, separating large data storage from metadata.
Related Classes/Methods:
This module provides a mechanism for associating arbitrary key-value metadata (tags) with workflow entities such as executions and jobs. These tags are persisted by the backend and can be used for organization, filtering, and querying historical workflow data, enhancing the "historical analysis capabilities."
Related Classes/Methods:
This module introduces the concept of "handles," which are Value objects that represent deferred or indirect access to other values. Handles are particularly useful for managing large or external data, allowing Redun to avoid unnecessary data movement until the value is actually needed, optimizing data flow.
Related Classes/Methods: