graph LR
Elephas_Keras_Model_Wrapper["Elephas Keras Model Wrapper"]
Spark_Distributed_Orchestrator["Spark Distributed Orchestrator"]
Elephas_Keras_Model_Wrapper -- "delegates tasks to" --> Spark_Distributed_Orchestrator
The Elephas Keras API subsystem is primarily defined by the elephas.ml_model and elephas.spark_model modules. These modules encapsulate the core logic for integrating Keras models with Apache Spark for distributed deep learning.
This component acts as the user-facing interface for integrating Keras models into the Elephas framework. It is responsible for wrapping standard Keras models, enabling their serialization and deserialization, and preparing them for distributed processing. It also provides high-level APIs for tasks such as feature extraction and prediction within the Elephas context, abstracting away the complexities of Spark integration from the Keras model definition.
Related Classes/Methods:
This is the core orchestration component responsible for managing and executing distributed deep learning tasks across the Spark cluster. It handles the distribution of Keras models and data partitions to worker nodes, initiates Spark jobs (e.g., training, prediction), and aggregates results (e.g., updated model weights, predictions) from the workers. It provides direct integration with Spark's RDD, MLlib, and ML APIs, serving as the bridge between the Keras model and the Spark distributed environment.
Related Classes/Methods: