Data_Management_Preprocessing.md

graph LR
    Specific_Dataset_Loaders["Specific Dataset Loaders"]
    Data_Caching_Manager["Data Caching Manager"]
    Transformation_Orchestrator["Transformation Orchestrator"]
    Core_Functional_Transformations["Core Functional Transformations"]
    Point_Cloud_Transformations["Point Cloud Transformations"]
    Specific_Dataset_Loaders -- "interacts with" --> Data_Caching_Manager
    Specific_Dataset_Loaders -- "feeds data into" --> Transformation_Orchestrator
    Data_Caching_Manager -- "interacts with" --> Specific_Dataset_Loaders
    Transformation_Orchestrator -- "orchestrates" --> Core_Functional_Transformations
    Transformation_Orchestrator -- "orchestrates" --> Point_Cloud_Transformations
    Core_Functional_Transformations -- "executed by" --> Transformation_Orchestrator
    Point_Cloud_Transformations -- "executed by" --> Transformation_Orchestrator

Loading

Details

The Data Management & Preprocessing subsystem is crucial for preparing diverse data types for consumption by machine learning models within CompressAI. It encompasses all stages from raw data acquisition to its transformation into a model-ready format.

Specific Dataset Loaders

Responsible for initializing, downloading (if necessary), and parsing various dataset types, acting as the primary interface for acquiring raw data. This component abstracts the complexities of different data sources.

Related Classes/Methods:

• compressai.datasets.pregenerated
• compressai.datasets.vimeo90k
• compressai.datasets.pointcloud.modelnet
• compressai.datasets.rawvideo
• compressai.datasets.pointcloud.s3dis
• compressai.datasets.pointcloud.semantic_kitti
• compressai.datasets.pointcloud.shapenet
• compressai.datasets.stack

Data Caching Manager

Manages the generation, loading, and retrieval of cached dataset information and individual data items. This component optimizes data access and reduces redundant processing, especially for large datasets, by persisting processed data.

Related Classes/Methods:

• compressai.datasets.cache

Transformation Orchestrator

Provides a unified, callable interface for applying a sequence of image and video transformations. It acts as the central entry point for defining and executing complex transformation pipelines, ensuring data consistency.

Related Classes/Methods:

• compressai.transforms.transforms
• compressai.registry.transforms

Core Functional Transformations

Implements fundamental, reusable transformation logic, primarily focusing on common operations like color space conversions (e.g., RGB to YUV) and sampling. These are the atomic, low-level units of transformation that can be composed.

Related Classes/Methods:

• compressai.transforms.functional

Point Cloud Transformations

Specializes in transformations specifically designed for point cloud data, including operations such as normalization, rotation, and other geometric manipulations. This component addresses the unique requirements of 3D data.

Related Classes/Methods:

• compressai.transforms.point

FAQ