graph LR
Executor["Executor"]
LocalExecutor["LocalExecutor"]
DockerExecutor["DockerExecutor"]
AWSBatchExecutor["AWSBatchExecutor"]
GCPBatchExecutor["GCPBatchExecutor"]
K8SExecutor["K8SExecutor"]
CommandUtils["CommandUtils"]
CodePackaging["CodePackaging"]
ScratchUtils["ScratchUtils"]
JobArrayer["JobArrayer"]
LocalExecutor -- "inherits from" --> Executor
LocalExecutor -- "uses" --> CommandUtils
LocalExecutor -- "uses" --> ScratchUtils
DockerExecutor -- "inherits from" --> Executor
DockerExecutor -- "uses" --> CommandUtils
DockerExecutor -- "uses" --> CodePackaging
AWSBatchExecutor -- "inherits from" --> Executor
AWSBatchExecutor -- "uses" --> DockerExecutor
AWSBatchExecutor -- "uses" --> JobArrayer
AWSBatchExecutor -- "uses" --> CodePackaging
GCPBatchExecutor -- "inherits from" --> Executor
GCPBatchExecutor -- "uses" --> DockerExecutor
GCPBatchExecutor -- "uses" --> JobArrayer
K8SExecutor -- "inherits from" --> Executor
K8SExecutor -- "uses" --> JobArrayer
K8SExecutor -- "uses" --> CodePackaging
DockerExecutor -- "uses" --> ScratchUtils
The Execution Backends subsystem in redun provides an extensible and unified interface for executing tasks across diverse computational environments. It abstracts the complexities of running tasks locally, within Docker containers, or on various cloud-specific services, ensuring a consistent workflow regardless of the underlying infrastructure.
This is the abstract base class for all executors in redun. It defines the common interface and lifecycle methods (e.g., submit, submit_script, start, stop, monitor) that all concrete executors must implement or override. It provides a foundational structure for managing job submissions, monitoring, and result processing, ensuring consistency across different execution environments.
Related Classes/Methods:
The simplest concrete implementation of Executor, designed for running tasks directly on the local machine where redun is invoked. It's primarily used for development, testing, and small-scale workflows, serving as the default execution backend.
Related Classes/Methods:
An executor that runs tasks within Docker containers. It handles Docker image management and container execution, serving as a crucial component for isolated and reproducible task execution. It can also act as a base for other container-orchestrated executors.
Related Classes/Methods:
A concrete implementation of Executor that leverages AWS Batch for executing tasks. It handles the specifics of interacting with AWS Batch services, including creating and managing job definitions, submitting jobs, and monitoring their status, integrating with AWS ECR and S3.
Related Classes/Methods:
Implements task execution on Google Cloud Platform's Batch service. Similar to AWSBatchExecutor, it manages GCP Batch jobs, handles resource allocation, and monitors job progress, often using DockerExecutor internally.
Related Classes/Methods:
This executor integrates with Kubernetes clusters to run tasks as Kubernetes Jobs. It manages Kubernetes resources (Jobs, Pods, Secrets) and handles the complexities of deploying and monitoring containerized tasks within a Kubernetes environment.
Related Classes/Methods:
A utility module for generating shell commands that execute redun tasks within the job environment. This is a fundamental component as all concrete executors rely on it to construct the actual commands that will be run.
Related Classes/Methods:
A utility module responsible for packaging the necessary redun code and user scripts into deployable artifacts (e.g., zip files, Docker images). This is crucial for remote executors to have access to the workflow code.
Related Classes/Methods:
A utility module for managing temporary files, storing job results, and handling error logs in a designated scratch space accessible by the executors. It ensures proper handling of intermediate and output data.
Related Classes/Methods:
A utility for handling job arrays, allowing a single job submission to represent multiple, similar tasks. This is particularly useful for optimizing and scaling batch processing on cloud platforms by reducing overhead.
Related Classes/Methods: