Skip to content

on_boarding.md

Latest commit

 

History

History
291 lines (241 loc) · 33.6 KB

on_boarding.md

File metadata and controls

291 lines (241 loc) · 33.6 KB
graph LR
    Crawler_Orchestration["Crawler Orchestration"]
    Execution_Engine["Execution Engine"]
    Request_Management["Request Management"]
    Downloader_System["Downloader System"]
    Scraping_Parsing["Scraping & Parsing"]
    Spider_Logic["Spider Logic"]
    Item_Processing_Pipelines["Item Processing Pipelines"]
    Configuration_Settings["Configuration & Settings"]
    Extensibility_Signals["Extensibility & Signals"]
    Crawler_Orchestration -- "initiates" --> Execution_Engine
    Execution_Engine -- "manages flow to" --> Request_Management
    Execution_Engine -- "manages flow to" --> Downloader_System
    Execution_Engine -- "manages flow to" --> Scraping_Parsing
    Request_Management -- "receives requests from" --> Execution_Engine
    Request_Management -- "provides requests to" --> Downloader_System
    Downloader_System -- "receives requests from" --> Execution_Engine
    Downloader_System -- "sends responses to" --> Execution_Engine
    Downloader_System -- "intercepts/modifies requests/responses between" --> Execution_Engine
    Scraping_Parsing -- "receives responses from" --> Execution_Engine
    Scraping_Parsing -- "sends items to" --> Item_Processing_Pipelines
    Scraping_Parsing -- "generates new requests back to" --> Execution_Engine
    Spider_Logic -- "generates initial requests for" --> Execution_Engine
    Spider_Logic -- "processes responses from" --> Scraping_Parsing
    Spider_Logic -- "produces items/requests for" --> Scraping_Parsing
    Item_Processing_Pipelines -- "receives items from" --> Scraping_Parsing
    Configuration_Settings -- "configures" --> Crawler_Orchestration
    Configuration_Settings -- "configures" --> Execution_Engine
    Configuration_Settings -- "configures" --> Request_Management
    Configuration_Settings -- "configures" --> Downloader_System
    Configuration_Settings -- "configures" --> Scraping_Parsing
    Configuration_Settings -- "configures" --> Spider_Logic
    Configuration_Settings -- "configures" --> Item_Processing_Pipelines
    Configuration_Settings -- "configures" --> Extensibility_Signals
    Extensibility_Signals -- "extends/communicates with" --> Crawler_Orchestration
    Extensibility_Signals -- "extends/communicates with" --> Execution_Engine
    Extensibility_Signals -- "extends/communicates with" --> Request_Management
    Extensibility_Signals -- "extends/communicates with" --> Downloader_System
    Extensibility_Signals -- "extends/communicates with" --> Scraping_Parsing
    Extensibility_Signals -- "extends/communicates with" --> Spider_Logic
    Extensibility_Signals -- "extends/communicates with" --> Item_Processing_Pipelines
    Extensibility_Signals -- "extends/communicates with" --> Configuration_Settings
    click Crawler_Orchestration href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Crawler Orchestration.md" "Details"
    click Execution_Engine href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Execution Engine.md" "Details"
    click Request_Management href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Request Management.md" "Details"
    click Downloader_System href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Downloader System.md" "Details"
    click Scraping_Parsing href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Scraping & Parsing.md" "Details"
    click Spider_Logic href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Spider Logic.md" "Details"
    click Item_Processing_Pipelines href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Item Processing Pipelines.md" "Details"
    click Configuration_Settings href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Configuration & Settings.md" "Details"
    click Extensibility_Signals href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/scrapy/Extensibility & Signals.md" "Details"
Loading

CodeBoardingDemoContact

Component Details

Scrapy is an open-source web crawling framework designed for fast, high-level screen scraping and web crawling. It allows users to define how to crawl websites and extract structured data using Spiders, which interact with a core Execution Engine that manages requests, responses, and item processing through a series of pluggable components like Downloader Middlewares and Item Pipelines. The framework is highly extensible via signals and extensions, and its behavior is controlled through a comprehensive settings system.

Crawler Orchestration

Manages the overall lifecycle of crawling processes, including initiating, executing, and terminating crawls. It provides the user interface for starting and managing Scrapy projects and individual spiders.

Related Classes/Methods:

Execution Engine

The central orchestrator of the Scrapy framework. It manages the flow of requests and responses between the Scheduler, Downloader, and Scraper, ensuring efficient and concurrent crawling operations.

Related Classes/Methods:

Request Management

Handles the queuing, prioritization, and deduplication of web requests. It ensures that only unique and relevant requests are processed by the Downloader, utilizing specific data structures for HTTP communication.

Related Classes/Methods:

Downloader System

Responsible for fetching web content from various sources, handling different protocols, and applying a chain of middlewares to requests before sending them and to responses upon receiving them.

Related Classes/Methods:

Scraping & Parsing

Processes downloaded responses, extracts structured data (items) based on spider logic, and can generate new requests for further crawling. It also includes tools for selecting data from HTML/XML.

Related Classes/Methods:

Spider Logic

Defines the custom crawling behavior for specific websites. Spiders specify how to initiate requests, parse responses, and yield extracted items or new requests. Spider middlewares allow for pre- and post-processing of spider input/output.

Related Classes/Methods:

Item Processing Pipelines

A series of components that process extracted items sequentially. This includes data validation, cleaning, persistence to various storage backends, and handling associated media files (images, files).

Related Classes/Methods:

Configuration & Settings

Centralized management of all configurable parameters for a Scrapy project. It allows users to customize the behavior of various components through a hierarchical settings system.

Related Classes/Methods:

  • scrapy.settings.Settings (full file reference)
  • scrapy.settings.BaseSettings (full file reference)
  • scrapy.utils.conf (full file reference)

Extensibility & Signals

Provides a flexible system for extending Scrapy's core functionality and enabling communication between different components through a robust signal dispatching mechanism. Extensions can hook into various stages of the crawling process.

Related Classes/Methods:

FAQ