# howie6879/ruia architecture

> Architecture diagram of the GitHub repository howie6879/ruia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/howie6879/ruia
- Repository: https://github.com/howie6879/ruia

## Overview

##### Project type and purpose
- Ruia is an **open‑source async Python web‑scraping micro‑framework** (not a monolithic application).
- It provides declarative spider classes, request/response abstractions, middleware/hooks, and item pipelines, all powered by `asyncio` and `aiohttp`.

##### File structure
- Top‑level directories:
- `ruia/`: the core framework modules (`spider.py`, `request.py`, `response.py`, `item.py`, `middleware.py`, `field.py`, `spider_hook.py`, utils, exceptions).
- `examples/`: sample spiders demonstrating usage.
- `docs/`: multilingual documentation, not part of runtime.
- `tests/`: unit tests for each component.
- Key files:
- `setup.py`, `Pipfile`: packaging and dependencies.
- `.travis.yml`: CI.
- `mkdocs.yml`: documentation build.

##### README for additional insights
- Technology stack: **Python 3.6+**, **asyncio**, **aiohttp**.
- Feature callouts: Middleware, Plugins, JavaScript support, Declarative Items.

##### Main components & interactions
a. **User‑Defined Spider**
- Subclasses `ruia.spider.Spider`.
- Defines `start_urls`, `parse` callbacks, `items` via `ruia.item.Item` + `ruia.field.Field`.
b. **Ruia Core**
- **Scheduler**: orchestrates URL queue, concurrency, and retry logic.
- **Network Layer**: uses `aiohttp.ClientSession` for HTTP requests.
- **Request/Response Abstraction**: wraps `aiohttp` requests into `ruia.request.Request` and responses into `ruia.response.Response`.
- **Middleware Layer**: pre‑ and post‑ request/response hooks (`ruia.middleware.Middleware`).
- **Spider Hooks**: custom lifecycle hooks (`ruia.spider_hook`).
- **Item Pipeline**: collects data via `Item` classes, applies field cleaning.
- **Utilities**: logging (`ruia.utils.log`), caching (optional), plugin registration.
c. **Asyncio Event Loop**
- Drives all concurrency: network calls, parsing, middleware, pipelines.
d. **External Services**
- Target Websites (the HTTP endpoints you crawl).

##### Architectural patterns & design principles
- **Asynchronous Event‑Driven**: everything runs on `asyncio`.
- **Pluggable Middleware/Hook**: separation of concerns for request lifecycle.
- **Declarative Data Modeling**: `Item` + `Field` akin to ORM schema.
- **Single‑Process Concurrency**: no threads; relies on non‑blocking I/O.

##### Details
- Be explicit about which module/file implements each component (e.g., Scheduler logic lives in `ruia.spider`).
- That `ruia.field` influences item parsing.
- That all operations occur under a single asyncio loop.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "User Layer"
        Spider["User-Defined Spider"]:::user
        Item["Item Definitions"]:::user
        Field["Field Definitions"]:::user
    end

    subgraph "Framework Layer"
        Scheduler["Scheduler"]:::core
        Req["Request Abstraction"]:::core
        Res["Response Abstraction"]:::core
        MW["Middleware Layer"]:::core
        Hooks["Spider Hooks"]:::core
        Pipeline["Item Pipeline"]:::core
        Exc["Exceptions"]:::core
        Utils["Utilities (Logging)"]:::core
    end

    EventLoop["asyncio Event Loop"]:::runtime

    subgraph "Network Layer"
        Client["aiohttp ClientSession"]:::network
        Target["Target Websites"]:::external
    end

    %% Data & Control Flow
    Spider -->|"start_urls, parse()"| Scheduler
    Scheduler -->|"schedule/await"| EventLoop
    EventLoop -->|"await/schedule"| Scheduler
    Scheduler -->|"Request()"| Req
    Req -->|"HTTP call"| Client
    Client -->|"GET/POST"| Target
    Target -->|"HTTP Response"| Res
    Res -->|"wrap Response"| MW
    MW -->|"pre/post hooks"| Spider
    Spider -->|"Parsed Items"| Pipeline
    Pipeline -->|"cleaned items"| Utils

    %% Extension Points
    Hooks ---|attach| Scheduler
    Hooks ---|attach| MW
    Hooks ---|attach| Pipeline
    Exc ---|error handling| Scheduler
    Exc ---|error handling| Req
    Exc ---|error handling| Res
    Exc ---|error handling| MW

    %% Component Mappings for click events
    click Spider "https://github.com/howie6879/ruia/tree/master/examples/"
    click Scheduler "https://github.com/howie6879/ruia/blob/master/ruia/spider.py"
    click Req "https://github.com/howie6879/ruia/blob/master/ruia/request.py"
    click Res "https://github.com/howie6879/ruia/blob/master/ruia/response.py"
    click MW "https://github.com/howie6879/ruia/blob/master/ruia/middleware.py"
    click Hooks "https://github.com/howie6879/ruia/blob/master/ruia/spider_hook.py"
    click Item "https://github.com/howie6879/ruia/blob/master/ruia/item.py"
    click Field "https://github.com/howie6879/ruia/blob/master/ruia/field.py"
    click Exc "https://github.com/howie6879/ruia/blob/master/ruia/exceptions.py"
    click Utils "https://github.com/howie6879/ruia/blob/master/ruia/utils/log.py"

    %% Tests Subgraph
    subgraph "Tests"
        FieldTest["Field Tests"]:::test
        ItemTest["Item Tests"]:::test
        MWTest["Middleware Tests"]:::test
        ReqTest["Request Tests"]:::test
        ResTest["Response Tests"]:::test
        SpiderTest["Spider Tests"]:::test
    end
    click FieldTest "https://github.com/howie6879/ruia/blob/master/tests/test_field.py"
    click ItemTest "https://github.com/howie6879/ruia/blob/master/tests/test_item.py"
    click MWTest "https://github.com/howie6879/ruia/blob/master/tests/test_middleware.py"
    click ReqTest "https://github.com/howie6879/ruia/blob/master/tests/test_request.py"
    click ResTest "https://github.com/howie6879/ruia/blob/master/tests/test_response.py"
    click SpiderTest "https://github.com/howie6879/ruia/blob/master/tests/test_spider.py"

    %% Styles
    classDef user fill:#FFD580,stroke:#333,stroke-width:1px
    classDef core fill:#B0E0E6,stroke:#333,stroke-width:1px
    classDef runtime fill:#98FB98,stroke:#333,stroke-width:1px
    classDef network fill:#AFEEEE,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef test fill:#F5DEB3,stroke:#333,stroke-width:1px
```
