# opennccu/nccucourse-crawler-by-andyjjrt architecture

> Architecture diagram of the GitHub repository opennccu/nccucourse-crawler-by-andyjjrt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opennccu/nccucourse-crawler-by-andyjjrt
- Repository: https://github.com/opennccu/nccucourse-crawler-by-andyjjrt

## Overview

##### Project type and purpose
- This is a Python-based CLI tool (“NCCUCourse”) rather than a full-stack web app.
- Its main purpose is:
- Authenticate to the NCCU iNCCU student portal
- Fetch course results, descriptions, rating data
- Translate rating text via Google Cloud
- Persist the cleaned and translated data locally (CSV/old JSON) or into a database

##### File structure
- Top-level scripts and modules:
- main.py – orchestrates the workflow
- fetchResult.py, fetchDescription.py, fetchRate.py – individual data-fetch steps
- translateRate.py – passes raw rating text to Google Translation API
- migrate.py – ingests old_data JSON into current storage format
- DB.py – encapsulates database / persistence logic
- User.py, util.py, constant.py – domain model and shared utilities
- Data directories:
- data/ – holds output CSV files for each semester
- old_data/ – legacy JSON dumps
- Configuration:
- .env – user credentials, YEAR/SEM, GOOGLE_APPLICATION_CREDENTIALS, OPENSSL_CONF
- openssl.conf – SSL settings for HTTPS

##### README for additional insights
- Specifies Python ≥3.9, uses pip requirements.
- Entry point `python main.py` (with possible `OPENSSL_CONF=…` shim).
- No built-in UI—pure CLI pipelines with HTTP, translation, and storage steps.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Environment & Configuration
    subgraph "Environment Loader"
        EnvDotEnv[".env"]:::internal
        Openssl["openssl.conf"]:::internal
    end

    %% Orchestrator
    Main["main.py"]:::internal

    %% Fetch Modules
    subgraph "Fetch Modules"
        FetchResult["fetchResult.py"]:::internal
        FetchDescription["fetchDescription.py"]:::internal
        FetchRate["fetchRate.py"]:::internal
    end

    %% Translation Module
    Translate["translateRate.py"]:::internal

    %% Persistence Layer
    subgraph "Persistence Layer"
        DB["DB.py"]:::internal
        DataStore["data/"]:::datastore
    end

    %% Utility & Domain
    subgraph "Utilities & Domain"
        UserModel["User.py"]:::internal
        Utils["util.py"]:::internal
        Constants["constant.py"]:::internal
    end

    %% Migration Module
    Migrate["migrate.py"]:::internal
    Legacy["old_data/"]:::datastore

    %% External Services
    subgraph "External Services"
        NCCU["NCCU iNCCU Portal"]:::external
        GCloud["Google Cloud Translate API"]:::external
    end

    %% Workflow
    Openssl -->|loads config| Main
    EnvDotEnv --> Main
    Main -->|login & scrape| FetchResult
    FetchResult -->|raw HTML| FetchDescription
    FetchDescription -->|JSON dicts| FetchRate
    FetchRate -->|raw ratings| Translate
    Translate -->|calls API| GCloud
    GCloud -->|translated text| Translate
    Translate -->|translated dicts| DB
    DB -->|writes CSV| DataStore

    %% Migration flow
    Legacy -->|old JSON| Migrate
    Migrate -->|normalized data| DB

    %% Utility dependencies
    Utils --> Main
    Constants --> Main
    UserModel --> FetchResult
    UserModel --> FetchDescription
    UserModel --> FetchRate
    UserModel --> Translate
    Utils --> DB
    Constants --> DB

    %% Click Events
    click Openssl "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/openssl.conf"
    click Main "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/main.py"
    click FetchResult "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/fetchResult.py"
    click FetchDescription "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/fetchDescription.py"
    click FetchRate "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/fetchRate.py"
    click Translate "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/translateRate.py"
    click DB "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/DB.py"
    click DataStore "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/tree/main/data/"
    click UserModel "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/User.py"
    click Utils "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/util.py"
    click Constants "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/constant.py"
    click Migrate "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/blob/main/migrate.py"
    click Legacy "https://github.com/opennccu/nccucourse-crawler-by-andyjjrt/tree/main/old_data/"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef external fill:#90EE90,stroke:#000,stroke-width:1px
    classDef datastore fill:#FFA500,stroke:#000,stroke-width:1px
```
