# quinnyen/2024_ewantcrawler architecture

> Architecture diagram of the GitHub repository quinnyen/2024_ewantcrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quinnyen/2024_ewantcrawler
- Repository: https://github.com/quinnyen/2024_ewantcrawler

## Overview

##### Project type and purpose
- This is a Python-based desktop data-scraping application with a GUI.
- Purpose: Automatically log into the “Ewant” course platform using Selenium, crawl course statistics (enrollment/pass counts), display progress in a UI, and export results to Excel.

##### File structure
- Top-level scripts & configs:
- main.py – application entry point
- build.py/.bat – build or packaging scripts
- requirements.txt – Python dependencies
- dependencies/dlls – external DLLs for packaging (e.g., cx_Freeze)
- Source code under src/:
- src/ui – UI layer (main_window.py)
- src/crawler – core crawling logic (login.py, parser.py, export.py)
- src/utils – utilities (config management, resource loading)
- No service-based or distributed modules—this is a single binary desktop application.

##### README for additional insights
- Uses Selenium to automate browser login and navigation.
- Outputs an Excel report.
- Reads/stores credentials via the OS keychain (handled in utils/config.py).
- UI allows keyword input, start/stop crawl, view logs, export report.

a. Main components
1) UI Component:
– Module: src/ui/main_window.py
– Technology: Tkinter or PyQt (inferred)
2) Crawler Component:
– Sub-components:
- Login Service (login.py)
- Parser Service (parser.py)
- Export Service (export.py)
– Technology: Selenium WebDriver
3) Utils Component:
– Config Manager (config.py) – handles keychain I/O
– Resource Loader (resource_utils.py)
4) External Systems:
– Ewant Web Platform (over HTTP/S via Selenium)
– OS Keychain (credential storage)
– File System (Excel output)
b. Relationships and interactions
- The UI triggers the Crawler Controller in src/crawler/export.py.
- Crawler Controller → Login Service → Selenium → Ewant platform
- On successful login → Parser Service navigates pages, collects data → returns data model
- UI listens for parser events and updates progress/log view
- When crawling finishes, UI invokes Export Service → writes .xlsx to File System
- Config Manager is invoked at startup to load saved credentials, and on exit to store new credentials.
c. Architectural patterns & design principles
- Clear separation of concerns: UI vs. business logic vs. utilities.
- Single-process, event-driven architecture: UI event loop drives crawling actions.
- Modular crawler services: each service class has one responsibility (login, parse, export).
d. Key technologies/frameworks
- Python 3.x runtime
- Selenium WebDriver (ChromeDriver, GeckoDriver, or similar)
- Excel export library (e.g., openpyxl or pandas)
- OS keychain interface (keyring library)
- GUI toolkit (Tkinter/PyQt)

##### Key points to capture
- Explicitly label modules with file names (e.g., src/crawler/login.py).
- Technology or library next to each component (e.g., “Selenium WebDriver”).
- Event flow (UI button click → crawl starts → progress events → export).
- One-way versus two-way interactions (e.g., UI polls parser events vs. parser pushes updates).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Project Setup"
        MainApp["main.py"]:::core
        BuildPy["build.py"]:::core
        BuildBat["build.bat"]:::core
        Requirements["requirements.txt"]:::core
        DLLs["dependencies/dlls"]:::core
        Resources["resources"]:::core
    end

    subgraph "UI Layer"
        UI["main_window.py\n(Tkinter/PyQt)"]:::ui
    end

    subgraph "Crawler Layer"
        Export["export.py\n(Controller & Export)"]:::crawler
        Login["login.py\n(Selenium WebDriver)"]:::crawler
        Parser["parser.py\n(Data Parser)"]:::crawler
    end

    subgraph "Utils Layer"
        Config["config.py\n(Keychain I/O)"]:::utils
        ResLoader["resource_utils.py"]:::utils
    end

    subgraph "External Systems"
        Ewant["Ewant Platform"]:::external
        Keychain["OS Keychain"]:::external
        FileSys["File System\n(Excel output)"]:::external
    end

    MainApp --> UI
    BuildPy --> UI
    BuildBat --> UI
    Requirements --> UI
    DLLs --> UI
    Resources --> UI

    UI -->|"Start Crawl"| Export
    Export -->|"login() call"| Login
    Login -->|"automate browser"| Ewant
    Ewant -->|"page HTML"| Parser
    Parser -->|"parsed data"| UI
    UI -->|"export report"| Export
    Export -->|"write .xlsx"| FileSys

    UI -->|"load/save creds"| Config
    Export -->|"load/save creds"| Config
    UI -->|"load resources"| ResLoader

    click MainApp "https://github.com/quinnyen/2024_ewantcrawler/blob/main/main.py"
    click BuildPy "https://github.com/quinnyen/2024_ewantcrawler/blob/main/build.py"
    click BuildBat "https://github.com/quinnyen/2024_ewantcrawler/blob/main/build.bat"
    click Requirements "https://github.com/quinnyen/2024_ewantcrawler/blob/main/requirements.txt"
    click DLLs "https://github.com/quinnyen/2024_ewantcrawler/tree/main/dependencies/dlls"
    click Resources "https://github.com/quinnyen/2024_ewantcrawler/tree/main/resources"
    click UI "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/ui/main_window.py"
    click Export "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/crawler/export.py"
    click Login "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/crawler/login.py"
    click Parser "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/crawler/parser.py"
    click Config "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/utils/config.py"
    click ResLoader "https://github.com/quinnyen/2024_ewantcrawler/blob/main/src/utils/resource_utils.py"

    classDef ui fill:#bbdefb,stroke:#0d47a1
    classDef crawler fill:#c8e6c9,stroke:#1b5e20
    classDef utils fill:#ffe0b2,stroke:#e65100
    classDef external fill:#e0e0e0,stroke:#424242
    classDef core fill:#f5f5f5,stroke:#9e9e9e
```
