graph LR
Faust_Topic["Faust Topic"]
Faust_Consumer["Faust Consumer"]
Faust_Producer["Faust Producer"]
Kafka_Broker["Kafka Broker"]
Faust_Application["Faust Application"]
Faust_Transport_Driver["Faust Transport Driver"]
Faust_Table_Manager["Faust Table Manager"]
Faust_Flow_Control["Faust Flow Control"]
Faust_Topic -- "delegates message sending to" --> Faust_Producer
Faust_Application -- "manages" --> Faust_Topic
Faust_Consumer -- "consumes messages from" --> Kafka_Broker
Faust_Consumer -- "manages offsets with" --> Kafka_Broker
Faust_Consumer -- "interacts with" --> Faust_Transport_Driver
Faust_Consumer -- "integrates with" --> Faust_Table_Manager
Faust_Consumer -- "integrates with" --> Faust_Flow_Control
Faust_Producer -- "publishes messages to" --> Kafka_Broker
Faust_Producer -- "manages delivery with" --> Kafka_Broker
Faust_Producer -- "interacts with" --> Faust_Transport_Driver
Faust_Application -- "manages" --> Faust_Consumer
Faust_Application -- "manages" --> Faust_Producer
Faust_Transport_Driver -- "communicates with" --> Kafka_Broker
Faust_Table_Manager -- "collaborates with" --> Faust_Consumer
Faust_Flow_Control -- "interacts with" --> Faust_Consumer
The Faust subsystem orchestrates real-time stream processing by interacting with an external Kafka Broker. The Faust Application serves as the central manager, configuring and overseeing Faust Topics, Faust Consumers, and Faust Producers. Faust Consumers are responsible for ingesting messages from the Kafka Broker, managing offsets, and integrating with Faust Table Manager for stateful processing and Faust Flow Control for backpressure. Conversely, Faust Producers publish processed events or new messages to the Kafka Broker, ensuring reliable delivery. Both consumers and producers abstract their low-level Kafka interactions through the Faust Transport Driver. This setup enables a robust, scalable, and fault-tolerant event-driven architecture.
Provides a declarative interface for defining Kafka topics, including their names, key/value types, and serialization/deserialization schemas. It represents the logical channels for event streams within the Faust application.
Related Classes/Methods:
Manages the consumption of messages from assigned Kafka partitions, handling offset management, group coordination, and deserialization of incoming event streams. It acts as the primary ingress point for data into the Faust application.
Related Classes/Methods:
Manages the sending of messages to Kafka topics, including message serialization, buffering, and flushing to the Kafka broker. It ensures reliable egress of processed events or new event streams generated by the application.
Related Classes/Methods:
The external Apache Kafka cluster that serves as the distributed, fault-tolerant message bus. It stores, replicates, and delivers event streams, acting as the central hub for all inter-component communication within the ecosystem.
Related Classes/Methods: None
The main orchestrator and lifecycle manager for a Faust stream processing application. It initializes, configures, and manages the various messaging components (topics, consumers, producers) and agents, binding them into a cohesive processing unit.
Related Classes/Methods:
Provides an abstract interface for low-level communication with the underlying messaging system (e.g., aiokafka for Kafka). It encapsulates the specific details of connecting to and interacting with the Kafka Broker for both message production and consumption.
Related Classes/Methods:
Manages the state recovery and consistency for Faust tables (state stores). It collaborates closely with the Faust Consumer to ensure that local application state is correctly synchronized with Kafka changelog topics, enabling fault-tolerant stateful processing.
Related Classes/Methods:
Regulates the rate of message consumption to prevent the Faust application from being overwhelmed by high message throughput. It implements backpressure mechanisms, pausing and resuming consumption to ensure stable and efficient operation.
Related Classes/Methods: