# wetbikeboy2500/web_content_parser architecture

> Architecture diagram of the GitHub repository wetbikeboy2500/web_content_parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wetbikeboy2500/web_content_parser
- Repository: https://github.com/wetbikeboy2500/web_content_parser

## Overview

##### Main Components

a) Core Components:
- Scraper System
- Parser System
- WQL (Web Query Language) Interpreter
- Headless Browser Interface
- Utility Layer

b) External Components:
- Mobile Headless Browser (flutter_inappwebview)
- Desktop Headless Browser (puppeteer)
- Target Websites

**┌─ Client Applications ─────────────────────────────────────┐**

- Uses web_content_parser_full.dart                    │

- ┤

- Parser System                           │
- ┤

- Scraper System                          │

- ┤

- WQL Interpreter    │    Headless Browser Interface    │

- ┘

##### Component Details

a) Parser System (Purple):
- JSON Models (Author, CatalogEntry, Chapter, etc.)
- Source Templates
- Compute Decorator
- ID Management System

b) Scraper System (Green):
- Subcomponents:
- Generic Scraper Functions
- WQL Functions
- Scraper Source Management
- Request Handling

c) WQL Interpreter (Blue):
- Following elements:
- Tokenizer
- Parser
- Statement Handlers
- Operators
- Logical Selectors

d) Headless Browser Interface (Orange):
- Include:
- Mobile Implementation
- Desktop Implementation
- Request Queue Management

##### Additional Elements

Special features:
- Result<T> wrapper flow
- Source YAML configuration flow
- ID/ChapterID management
- Compute decoration pattern

##### The extensibility points

Keep the abstraction level high enough to show the architectural decisions while including enough detail to understand the unique features like WQL and the Result pattern.

Remember to show the relationship between the three main imports (util.dart, scraper.dart, and parser.dart) and how they can be used independently or together through web_content_parser_full.dart.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client["Client Applications"]
        WCP["Web Content Parser Full"]:::main
    end

    subgraph Parser["Parser System"]
        PM["JSON Models"]:::parser
        ST["Source Templates"]:::parser
        CD["Compute Decorator"]:::parser
        IMS["ID Management System"]:::parser
    end

    subgraph Scraper["Scraper System"]
        GSF["Generic Scraper Functions"]:::scraper
        WQLF["WQL Functions"]:::scraper
        SSM["Scraper Source Management"]:::scraper
        RH["Request Handling"]:::scraper
    end

    subgraph WQL["WQL Interpreter"]
        TK["Tokenizer"]:::wql
        PR["Parser"]:::wql
        SH["Statement Handlers"]:::wql
        OP["Operators"]:::wql
        LS["Logical Selectors"]:::wql
    end

    subgraph HBI["Headless Browser Interface"]
        MI["Mobile Implementation"]:::headless
        DI["Desktop Implementation"]:::headless
        RQ["Request Queue Management"]:::headless
    end

    subgraph Util["Utility Layer"]
        RES["Result Handling"]:::util
        RT["Request Types"]:::util
        YP["YAML Parsing"]:::util
        LOG["Logging"]:::util
    end

    WCP --> PM
    WCP --> GSF
    WCP --> RES

    PM --> CD
    PM --> IMS
    ST --> CD
    
    GSF --> WQLF
    WQLF --> SSM
    SSM --> RH
    
    WQLF --> TK
    TK --> PR
    PR --> SH
    PR --> OP
    PR --> LS
    
    RH --> MI
    RH --> DI
    MI --> RQ
    DI --> RQ

    classDef main fill:#2D3748,stroke:#2D3748,color:white
    classDef parser fill:#805AD5,stroke:#805AD5,color:white
    classDef scraper fill:#38A169,stroke:#38A169,color:white
    classDef wql fill:#3182CE,stroke:#3182CE,color:white
    classDef headless fill:#DD6B20,stroke:#DD6B20,color:white
    classDef util fill:#718096,stroke:#718096,color:white

    click WCP "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/web_content_parser_full.dart"
    click PM "https://github.com/Wetbikeboy2500/web_content_parser/tree/main/lib/src/parser/json/"
    click ST "https://github.com/Wetbikeboy2500/web_content_parser/tree/main/lib/src/parser/sources/"
    click CD "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/parser/sources/computeDecorator.dart"
    click GSF "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/scraper/generic/scrapeFunctions.dart"
    click WQLF "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/scraper/wql/wqlFunctions.dart"
    click SSM "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/scraper/scraperSource.dart"
    click TK "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/wql/wql.dart"
    click PR "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/wql/interpreter/interpreter.dart"
    click SH "https://github.com/Wetbikeboy2500/web_content_parser/tree/main/lib/src/wql/statements/"
    click OP "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/wql/suboperations/operator.dart"
    click LS "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/wql/suboperations/logicalSelector.dart"
    click MI "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/packages/web_content_headless/lib/src/mobileHeadless.dart"
    click DI "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/packages/web_content_headless/lib/src/desktopHeadless.dart"
    click RES "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/util/Result.dart"
    click RT "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/util/RequestType.dart"
    click YP "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/util/parseYaml.dart"
    click LOG "https://github.com/Wetbikeboy2500/web_content_parser/blob/main/lib/src/util/log.dart"
```
