LLM_Processing_Nodes.md

graph LR
    BaseNode["BaseNode"]
    GenerateAnswerNode["GenerateAnswerNode"]
    GenerateCodeNode["GenerateCodeNode"]
    PromptRefinerNode["PromptRefinerNode"]
    GenerateAnswerFromImageNode["GenerateAnswerFromImageNode"]
    External_LLM_Service["External LLM Service"]
    Internal_Code_Generation_Validation_Logic["Internal Code Generation & Validation Logic"]
    Internal_Syntax_Checker["Internal Syntax Checker"]
    Internal_Sandbox_Executor["Internal Sandbox Executor"]
    Internal_Output_Validator["Internal Output Validator"]
    Internal_Semantic_Comparator["Internal Semantic Comparator"]
    Internal_Image_Processor["Internal Image Processor"]
    GenerateAnswerNode -- "inherits from" --> BaseNode
    GenerateCodeNode -- "inherits from" --> BaseNode
    PromptRefinerNode -- "inherits from" --> BaseNode
    GenerateAnswerFromImageNode -- "inherits from" --> BaseNode
    GenerateAnswerNode -- "invokes" --> External_LLM_Service
    GenerateCodeNode -- "interacts with" --> External_LLM_Service
    GenerateCodeNode -- "orchestrates" --> Internal_Code_Generation_Validation_Logic
    Internal_Code_Generation_Validation_Logic -- "includes" --> Internal_Syntax_Checker
    Internal_Code_Generation_Validation_Logic -- "includes" --> Internal_Sandbox_Executor
    Internal_Code_Generation_Validation_Logic -- "includes" --> Internal_Output_Validator
    Internal_Code_Generation_Validation_Logic -- "includes" --> Internal_Semantic_Comparator
    PromptRefinerNode -- "prepares prompts for" --> External_LLM_Service
    GenerateAnswerFromImageNode -- "processes images using" --> Internal_Image_Processor
    GenerateAnswerFromImageNode -- "generates answers through" --> External_LLM_Service

Loading

Details

The scrapegraphai.nodes subsystem is built around a BaseNode abstract class, providing a common interface for various LLM-driven processing nodes. Key components include GenerateAnswerNode for text-based answer generation, GenerateCodeNode for iterative code generation and validation, PromptRefinerNode for optimizing LLM prompts, and GenerateAnswerFromImageNode for multi-modal image analysis. The External LLM Service represents the integration point with various Large Language Models, which are leveraged by GenerateAnswerNode, GenerateCodeNode, and GenerateAnswerFromImageNode for their core functionalities. GenerateCodeNode internally orchestrates a sophisticated Internal Code Generation & Validation Logic which encompasses Internal Syntax Checker, Internal Sandbox Executor, Internal Output Validator, and Internal Semantic Comparator to ensure robust code output. Similarly, GenerateAnswerFromImageNode utilizes an Internal Image Processor for handling image inputs before LLM interaction. This architecture emphasizes modularity and specialized processing capabilities for different data types and tasks within the scraping and AI pipeline.

BaseNode

The foundational abstract class for all LLM-driven processing nodes. It provides common initialization, input key parsing, and expression evaluation functionalities, ensuring a consistent interface and shared foundational logic across specialized nodes.

Related Classes/Methods:

• scrapegraphai.nodes.base_node:BaseNode

GenerateAnswerNode

Specializes in generating concise and relevant textual answers from input data by querying an external Large Language Model. This is a core AI-powered data extraction capability.

Related Classes/Methods:

• scrapegraphai.nodes.generate_answer_node:GenerateAnswerNode

GenerateCodeNode

Focuses on generating and iteratively refining executable code, incorporating robust validation mechanisms. It ensures the correctness and functionality of generated code through syntax checking, sandbox execution, output validation, and semantic comparison.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode

PromptRefinerNode

Dedicated to optimizing and refining input prompts before they are sent to an LLM. This process enhances the clarity, specificity, and effectiveness of prompts, leading to more accurate and desirable LLM responses.

Related Classes/Methods:

• scrapegraphai.nodes.prompt_refiner_node:PromptRefinerNode

GenerateAnswerFromImageNode

Handles the processing of image inputs to generate textual answers, leveraging multi-modal capabilities of LLMs or integrated image-to-text functionalities.

Related Classes/Methods:

• scrapegraphai.nodes.generate_answer_from_image_node:GenerateAnswerFromImageNode

External LLM Service

An external service providing Large Language Model capabilities (e.g., OpenAI, Groq, Azure, Gemini, Ollama) used by the processing nodes for answer generation, code generation, and multi-modal tasks.

Related Classes/Methods:

• scrapegraphai.llms.base.BaseLLM

Internal Code Generation & Validation Logic

Internal logic for syntax checking, sandbox execution, output validation, and semantic comparison within GenerateCodeNode.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode.overall_reasoning_loop

Internal Syntax Checker

Component for syntax checking within GenerateCodeNode's internal logic.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode.syntax_check

Internal Sandbox Executor

Component for sandbox execution within GenerateCodeNode's internal logic.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode.create_sandbox_and_execute

Internal Output Validator

Component for output validation within GenerateCodeNode's internal logic.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode.validate_dict

Internal Semantic Comparator

Component for semantic comparison within GenerateCodeNode's internal logic.

Related Classes/Methods:

• scrapegraphai.nodes.generate_code_node:GenerateCodeNode.semantic_comparison

Internal Image Processor

Internal component for processing images within GenerateAnswerFromImageNode.

Related Classes/Methods:

• scrapegraphai.nodes.generate_answer_from_image_node:GenerateAnswerFromImageNode.process_image

FAQ