# leduonng/crawl-truyenfull architecture

> Architecture diagram of the GitHub repository leduonng/crawl-truyenfull, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leduonng/crawl-truyenfull
- Repository: https://github.com/leduonng/crawl-truyenfull

## Overview

##### And Illustrate the Main Components
- User Input: Label a component showing how users invoke the crawl either via the command line (python main.py with URL) or via run.bat (which prompts for the URL).
- Crawler Orchestrator: This is the central component in main.py that coordinates the crawling process.
- Chapter Discovery Module: Represent a subcomponent that scans the pagination to determine the total number of chapters. Note in the README that it looks for the “Cuối” link or the largest chapter number.
- Content Parser/Cleaner: Include a module that applies BeautifulSoup to clean the HTML, removes ads, and ensures only the required content is extracted.
- Storage Manager: This module handles file writing:
- A File Saver for the complete story as a Word document (using python-docx).
- A File Splitter to produce segmented text files (each ~18000 words).
- Logging Mechanism: A component dedicated to logging each step of the process into the “logs” directory, capturing errors and progress.

##### Relationships and Data Flow
- The Orchestrator then delegates the job to the Chapter Discovery module to identify the chapter range.
- Once chapters are identified, the Multi-threaded Crawling Engine concurrently fetches content by interacting with the Content Parser module.
- Each fetched chapter's content flows to the Storage Manager, which saves the data into the corresponding file formats.
- Parallelly, all major steps and errors are reported to the Logging Mechanism.

##### Key Architectural Patterns and Design Principles
- Modularity: Each of the above components is logically separated and responsible for a specific task, which makes the system easier to manage and extend.
- Resilience: Incorporate error handling and retry logic. Illustrate how the system continues despite encountering empty chapters or errors.
- Batch Processing: Batch handling (crawling 10 chapters per batch) is an important internal detail that can be noted.

##### Relevant Technologies and Frameworks
- Language: Python 3.6+
- HTTP Requests: The “requests” library for fetching web content.
- HTML Parsing: “beautifulsoup4” for processing and cleaning page content.
- Document Generation: “python-docx” for creating Word documents.
- Operating Environment: Windows is indirectly referenced by run.bat.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interaction"
        UI("User Input (run.bat & main.py)"):::userComponent
    end

    subgraph "Crawling Core"
        CO("Crawler Orchestrator"):::orchestrator
        CD("Chapter Discovery Module"):::chapter
        MTCE("Multi-threaded Crawling Engine"):::mtoc
        CP("Content Parser/Cleaner"):::content
        SM("Storage Manager"):::storage
    end

    L("Logging Mechanism"):::logging

    UI -->|"providesURL"| CO
    CO -->|"discoversChapters"| CD
    CO -->|"initiatesCrawling"| MTCE
    MTCE -->|"fetchesChapters"| CP
    CP -->|"deliversCleanedContent"| SM
    MTCE -->|"reportsErrorRetry"| L
    CP -->|"logsParsing"| L
    SM -->|"logsStorage"| L

    click UI "https://github.com/leduonng/crawl-truyenfull/blob/main/run.bat"
    click UI "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"
    click CO "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"
    click CD "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"
    click MTCE "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"
    click CP "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"
    click SM "https://github.com/leduonng/crawl-truyenfull/blob/main/main.py"

    classDef userComponent fill:#87CEEB,stroke:#000,stroke-width:2px;
    classDef orchestrator fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef chapter fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef mtoc fill:#9370DB,stroke:#000,stroke-width:2px;
    classDef content fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef storage fill:#40E0D0,stroke:#000,stroke-width:2px;
    classDef logging fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
