Skip to content

on_boarding.md

Latest commit

 

History

History
92 lines (56 loc) · 7.23 KB

on_boarding.md

File metadata and controls

92 lines (56 loc) · 7.23 KB
graph LR
    CLI_Interface["CLI Interface"]
    Ingest_Orchestrator["Ingest Orchestrator"]
    Message_Broker_Celery_Redis_["Message Broker (Celery/Redis)"]
    Ingest_Sync_Workers["Ingest/Sync Workers"]
    iRODS_Service_Layer["iRODS Service Layer"]
    Persistence_Layer_Redis_["Persistence Layer (Redis)"]
    CLI_Interface -- "initiates" --> Ingest_Orchestrator
    Ingest_Orchestrator -- "manages job state via" --> Persistence_Layer_Redis_
    Ingest_Orchestrator -- "enqueues tasks to" --> Message_Broker_Celery_Redis_
    Message_Broker_Celery_Redis_ -- "distributes tasks to" --> Ingest_Sync_Workers
    Ingest_Sync_Workers -- "performs operations via" --> iRODS_Service_Layer
    Ingest_Sync_Workers -- "reports status to" --> Ingest_Orchestrator
    click CLI_Interface href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/CLI_Interface.md" "Details"
    click Ingest_Orchestrator href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/Ingest_Orchestrator.md" "Details"
    click Message_Broker_Celery_Redis_ href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/Message_Broker_Celery_Redis_.md" "Details"
    click Ingest_Sync_Workers href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/Ingest_Sync_Workers.md" "Details"
    click iRODS_Service_Layer href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/iRODS_Service_Layer.md" "Details"
    click Persistence_Layer_Redis_ href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/irods_capability_automated_ingest/Persistence_Layer_Redis_.md" "Details"
Loading

CodeBoardingDemoContact

Details

The irods_capability_automated_ingest system is designed as a distributed, asynchronous data ingest and synchronization framework. Users interact with the system via the CLI Interface, which sends commands to the Ingest Orchestrator. The Orchestrator is the central intelligence, managing job states through the Persistence Layer (Redis) and delegating work by enqueuing tasks to the Message Broker (Celery/Redis). A pool of Ingest/Sync Workers retrieves these tasks from the Message Broker and executes the actual data transfer or deletion operations, interacting with the iRODS data management system through the iRODS Service Layer. This architecture ensures scalability, reliability, and extensibility, allowing for various data sources and custom event handling throughout the synchronization process.

CLI Interface [Expand]

The user-facing command-line tool for initiating, configuring, and monitoring data ingest and synchronization jobs.

Related Classes/Methods:

Ingest Orchestrator [Expand]

The central control component managing the entire lifecycle of data ingest and synchronization jobs.

Related Classes/Methods:

Message Broker (Celery/Redis) [Expand]

An external, distributed message queue system for asynchronous communication and reliable task distribution.

Related Classes/Methods:

Ingest/Sync Workers [Expand]

A pool of distributed worker processes that consume tasks from the Message Broker and execute data synchronization and ingest operations.

Related Classes/Methods:

iRODS Service Layer [Expand]

Provides a unified and abstracted interface for all interactions with the iRODS data management system.

Related Classes/Methods:

Persistence Layer (Redis) [Expand]

Utilizes Redis as a fast, in-memory key-value store for persisting application state, particularly for job management.

Related Classes/Methods:

FAQ