graph LR
Dataflow["Dataflow"]
fn["fn"]
operator["operator"]
_Scope["_Scope"]
_with_scope["_with_scope"]
_to_ref["_to_ref"]
_gen_op_cls["_gen_op_cls"]
_anno_to_typ["_anno_to_typ"]
Dataflow -- "incorporates operations defined by" --> fn
Dataflow -- "integrates custom operators defined by" --> operator
Dataflow -- "utilizes" --> _Scope
Dataflow -- "uses" --> _with_scope
fn -- "calls" --> _to_ref
fn -- "uses" --> _anno_to_typ
fn -- "uses" --> _with_scope
_with_scope -- "manages the state of" --> _Scope
_gen_op_cls -- "uses" --> _anno_to_typ
The Dataflow Definition API subsystem is responsible for providing the programmatic interface for users to construct and define real-time data processing pipelines within the Bytewax framework. It focuses on the declarative definition of dataflow graphs, allowing users to specify data sources, transformations, and sinks.
Serves as the top-level container and builder for a stream processing pipeline. Users instantiate Dataflow to begin defining their pipeline, adding various operations to it. It manages the overall structure and state of the dataflow graph, acting as the conceptual representation of the entire pipeline.
Related Classes/Methods:
Provides a straightforward, user-facing function/decorator for defining individual, atomic dataflow steps or transformations. It allows users to specify the processing logic for data by wrapping a Python function into a dataflow operation.
Related Classes/Methods:
A user-facing decorator that enables the definition of custom, reusable dataflow operators. It allows users to encapsulate a sequence of operations or a more complex processing logic into a single, reusable component, extending the framework's built-in capabilities.
Related Classes/Methods:
Manages the hierarchical context and nesting of operations within the dataflow, crucial for maintaining structural integrity and enabling nested dataflow definitions.
Related Classes/Methods:
A context manager that facilitates entering and exiting specific operational scopes, ensuring correct context for dataflow operations and proper graph construction within nested structures.
Related Classes/Methods:
Converts various dataflow elements (like ports and IDs) into internal references, essential for building the dataflow's directed acyclic graph (DAG) representation and linking operations.
Related Classes/Methods:
Dynamically generates the Python class representation for a dataflow operator, based on user definitions and type annotations, enabling the runtime creation of operator types.
Related Classes/Methods:
Processes Python type annotations to determine the types of inputs and outputs for dataflow operations and operators, crucial for type checking and internal graph validation.
Related Classes/Methods: