graph LR
Source["Source"]
SourceManager["SourceManager"]
KafkaSource["KafkaSource"]
FileSource["FileSource"]
KinesisSource["KinesisSource"]
PubSubSource["PubSubSource"]
FileFetcher["FileFetcher"]
KinesisConsumer["KinesisConsumer"]
PubSubConsumer["PubSubConsumer"]
KafkaSource -- "inherits from" --> Source
FileSource -- "inherits from" --> Source
KinesisSource -- "inherits from" --> Source
PubSubSource -- "inherits from" --> Source
SourceManager -- "manages" --> Source
FileSource -- "uses" --> FileFetcher
KinesisSource -- "uses" --> KinesisConsumer
PubSubSource -- "uses" --> PubSubConsumer
The quixstreams.sources subsystem is responsible for ingesting data from various external systems into the Quix Streams framework. It provides a flexible and extensible architecture through an abstract Source component, which defines a common interface for all data input mechanisms. The SourceManager orchestrates the lifecycle and management of these diverse data sources. Concrete implementations like KafkaSource, FileSource, KinesisSource, and PubSubSource handle specific data platforms, often leveraging dedicated consumer or fetcher components for direct interaction with their respective APIs. This design ensures modularity and allows for easy integration of new data sources.
This abstract base class defines the common interface and contract for all data ingestion mechanisms within the framework. It ensures a consistent API for different types of data sources.
Related Classes/Methods:
Acts as the orchestrator for all data sources, managing their lifecycle, including starting, stopping, configuration, and handling exceptions. It provides a centralized control point for data ingress.
Related Classes/Methods:
A primary concrete implementation responsible for ingesting messages directly from Apache Kafka topics. It translates raw Kafka messages into the framework's internal data format for processing.
Related Classes/Methods:
Provides a generalized approach for ingesting data from various file storage systems (e.g., local files, cloud storage). It internally utilizes a FileFetcher utility for efficient file retrieval.
Related Classes/Methods:
quixstreams.sources.community.file.base.FileSource:23-204quixstreams.sources.community.file.components.file_fetcher.FileFetcher:11-59
A concrete implementation for ingesting real-time data streams from Amazon Kinesis. It leverages a KinesisConsumer component for direct interaction with the Kinesis API.
Related Classes/Methods:
quixstreams.sources.community.kinesis.kinesis.KinesisSource:22-160quixstreams.sources.community.kinesis.consumer.KinesisConsumer:61-190
A concrete implementation for ingesting real-time data streams from Google Cloud Pub/Sub. It uses a PubSubConsumer component to manage interactions with the Google Cloud Pub/Sub API.
Related Classes/Methods:
quixstreams.sources.community.pubsub.pubsub.PubSubSource:20-153quixstreams.sources.community.pubsub.consumer.PubSubConsumer:32-185
Utility for efficient file retrieval, used by FileSource.
Related Classes/Methods:
Component for direct interaction with the Kinesis API, used by KinesisSource.
Related Classes/Methods:
Component to manage interactions with the Google Cloud Pub/Sub API, used by PubSubSource.
Related Classes/Methods: