# heyihuang826/ncku_course architecture

> Architecture diagram of the GitHub repository heyihuang826/ncku_course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/heyihuang826/ncku_course
- Repository: https://github.com/heyihuang826/ncku_course

## Overview

This repository is a Python-based batch scraper and data export pipeline for NCKU course information.

#### High-level architecture

The project is organized as a small command-line/job runner with three major concerns:

1. **Data acquisition from the NCKU course site**
2. **Captcha solving and authentication support**
3. **Post-processing into Excel and optional delivery to OneDrive or local storage**

The system is designed to run either:
- manually on a local machine, or
- automatically through GitHub Actions on a schedule.

#### Main execution path

The runtime entry point appears to be:

- `src/main.py` as the top-level launcher
- `src/ncku_course/run.py` as the orchestration layer

The core job flow is:

1. Start the run.
2. Authenticate or establish the session needed for the NCKU site.
3. Crawl course data, likely using multithreaded requests.
4. Use the captcha model when the site requires verification.
5. Transform the collected data into an Excel workbook.
6. Save the file locally, upload it to OneDrive, or possibly persist it in the repository/GitHub Actions context.

#### Subsystems

##### 1) Crawling subsystem
Located mainly in:

- `src/ncku_course/crawler.py`

This is the data collection boundary around the external NCKU website. Based on the README, it is responsible for:
- fetching course listings / course details
- handling repeated scraping at scale
- using multithreading to improve throughput

This subsystem is the main consumer of the site’s public or session-protected endpoints.

##### 2) Authentication and captcha subsystem
Located in:

- `src/ncku_course/authenticate.py`
- `src/ncku_course/captcha/`

This is an important architectural boundary because the site can block scraping without verification.

The captcha package contains:
- `captcha/main.py`
- `captcha/predict.py`
- a trained model artifact (`model_20240828_2152.h5` and SavedModel files)

This indicates the repository bundles a local ML-based captcha solver rather than relying on a third-party captcha service. Architecturally, this subsystem sits between the crawler and the website session flow:
- crawler hits protected pages
- captcha/authentication layer solves or bypasses the verification step
- session proceeds with authenticated requests

The model files are a deployment dependency, not just source code, so the captcha solver is part of the runtime environment.

##### 3) Excel generation subsystem
Located in:

- `src/ncku_course/excel.py`

This subsystem converts scraped course data into a structured Excel file. It is the output formatting boundary for the whole application. The README’s example file suggests the output is a finished report workbook rather than raw CSV or JSON.

This layer likely depends on the data shape produced by the crawler and is responsible for organizing the final dataset into a user-consumable spreadsheet.

##### 4) OneDrive integration subsystem
Located in:

- `src/ncku_course/onedrive/`

This is a self-contained integration boundary for cloud storage. It includes:
- API interaction code
- token handling
- key/credential logic
- upload helpers
- custom exceptions and shared utilities

From the repository structure, this looks like a dedicated client wrapper around OneDrive rather than ad hoc upload code. That separation matters architecturally because the core scraping pipeline does not need to know OneDrive protocol details; it only hands off the generated Excel file.

##### 5) Shared utilities and configuration
Located in:

- `src/ncku_course/_inner_utils.py`
- `src/ncku_course/const.py`
- `src/ncku_course/onedrive/_internal_utils.py`

These files likely hold cross-cutting helpers, constants, and shared internal logic used by multiple subsystems.

They are not primary business logic boundaries, but they support:
- request/session handling
- file naming or path conventions
- reusable API helpers
- token/secret-related plumbing

#### Data flow

A typical run follows this path:

1. **Start / orchestration**
   - `main.py` or `run.py` initializes the job and chooses the storage target.

2. **Session preparation**
   - authentication logic prepares access to the course site.
   - if captcha appears, the captcha model is used.

3. **Scraping**
   - the crawler collects course data, likely in parallel threads.

4. **Normalization**
   - the raw scraped records are shaped into a consistent in-memory representation.

5. **Excel export**
   - the workbook is generated.

6. **Delivery**
   - either:
     - upload to OneDrive, or
     - save locally
   - in GitHub Actions mode, the workflow can also preserve the file in the repo-based automation flow, depending on configuration.

#### External systems and boundaries

##### NCKU course website
This is the primary upstream data source. The whole repository exists to interact with it reliably over time.

##### OneDrive
Optional downstream storage target. It is used as a cloud sink for the generated Excel file.

##### GitHub Actions
The repository includes `.github/workflows/main.yml`, which indicates scheduled automation. This turns the scraper into a periodic batch job running in CI rather than only a user-invoked tool.

The workflow boundary matters because the project is meant to run:
- without human interaction,
- on a schedule,
- with secrets/configuration injected by the CI environment.

#### Notable architectural characteristics

- **Batch-oriented, not request/response**
  - This is a scheduled data pipeline, not a web server or interactive app.

- **Anti-blocking emphasis**
  - Captcha-solving is a first-class subsystem, showing the scraper is designed to survive defensive website behavior.

- **Separation of concerns**
  - crawling, captcha/auth, Excel output, and OneDrive upload are isolated into distinct packages/modules.

- **Portable output**
  - The final artifact is an Excel file, making the repository useful both for automated distribution and manual inspection.

- **Operational flexibility**
  - Supports local execution, GitHub Actions execution, and different storage destinations.

#### Technology/runtime implications

The project targets:
- **Python 3.8.10** as stated in the README
- a machine-learning captcha model packaged with the repo
- GitHub Actions for scheduled automation
- OneDrive integration for optional cloud delivery

#### Suggested diagram boundaries

If you were drawing the architecture, the cleanest boundaries would be:

- **Job entry / orchestration**
  - `main.py`, `run.py`

- **NCKU site access**
  - `authenticate.py`, `crawler.py`, captcha package

- **Data transformation**
  - `excel.py`

- **Artifact delivery**
  - `onedrive/` plus local file output

- **External services**
  - NCKU website
  - OneDrive
  - GitHub Actions scheduler/runtime

In short, this repo is a scheduled scraping pipeline with ML-assisted captcha handling and flexible Excel output delivery.

Main entry points: [`src/ncku_course/onedrive/api.py`](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/api.py) (OneDrive), [`src/ncku_course/run.py`](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/run.py) (Run flow), [`src/ncku_course/authenticate.py`](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/authenticate.py) (Auth), [`src/ncku_course/excel.py`](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/excel.py) (Excel)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Job runner"
  n1(("Main<br/>entrypoint<br/>[main.py]"))
  n2["Run flow<br/>orchestrator<br/>[run.py]"]
  n3["Consts<br/>shared config<br/>[const.py]"]
  n4["Utils<br/>shared helpers<br/>[_inner_utils.py]"]
end

subgraph "NCKU access"
  n5["Auth<br/>[authenticate.py]"]
  n6["Crawler<br/>scraper<br/>[crawler.py]"]
  n7["Captcha<br/>ml solver<br/>[main.py]"]
  n8["Predict<br/>inference<br/>[predict.py]"]
  n9[("Model")]
end

subgraph "Output delivery"
  n10["Excel<br/>exporter<br/>[excel.py]"]
  n11["OneDrive<br/>cloud client<br/>[api.py]"]
  n12["Upload<br/>delivery<br/>[upload.py]"]
  n13["Tokens<br/>credentials<br/>[tokens.py]"]
  n14["Data<br/>payloads<br/>[data.py]"]
  n15["O-Utils<br/>client helpers<br/>[_internal_utils.py]"]
  n16["Keys<br/>secret helpers<br/>[key.py]"]
  n17["Errors<br/>exceptions<br/>[exceptions.py]"]
end

subgraph "External systems"
  n18["GitHub Actions<br/>scheduler<br/>[main.yml]"]
  n19(("NCKU site<br/>upstream service"))
  n20(("OneDrive cloud<br/>downstream service"))
end

n1 -->|"starts"| n2
n2 -->|"prepares"| n5
n2 -->|"collects"| n6
n2 -->|"exports"| n10
n2 -->|"delivers"| n12
n5 -->|"handles captcha"| n7
n7 -->|"runs inference"| n8
n8 -->|"loads model"| n9
n6 -->|"requests data"| n19
n5 -->|"authenticates"| n19
n10 -->|"uses"| n3
n10 -->|"uses"| n4
n12 -->|"calls"| n11
n11 -->|"manages"| n13
n11 -->|"uses"| n14
n11 -->|"uses"| n15
n11 -->|"uses"| n16
n11 -->|"raises"| n17
n12 -->|"uploads to"| n20
n18 -.->|"runs"| n1
n18 -.->|"orchestrates"| n2

click n1 "https://github.com/heyihuang826/ncku_course/blob/master/src/main.py"
click n2 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/run.py"
click n3 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/const.py"
click n4 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/_inner_utils.py"
click n5 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/authenticate.py"
click n6 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/crawler.py"
click n7 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/captcha/main.py"
click n8 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/captcha/predict.py"
click n9 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/captcha/model_20240828_2152.h5"
click n10 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/excel.py"
click n11 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/api.py"
click n12 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/upload.py"
click n13 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/tokens.py"
click n14 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/data.py"
click n15 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/_internal_utils.py"
click n16 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/key.py"
click n17 "https://github.com/heyihuang826/ncku_course/blob/master/src/ncku_course/onedrive/exceptions.py"
click n18 "https://github.com/heyihuang826/ncku_course/blob/master/.github/workflows/main.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class n1,n2,n3,n4 toneBlue
class n5,n6,n7,n8,n9 toneAmber
class n10,n11,n12,n13,n14,n15,n16,n17 toneMint
class n18,n19,n20 toneRose
```

## Components

### Job runner

Entry points and scheduled execution for the batch pipeline.

- [Main](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/main.py): entrypoint · Top-level launcher for local or CI execution. · `src/main.py`
- [Run flow](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/run.py): orchestrator · Coordinates the end-to-end batch job. · `src/ncku_course/run.py`
- [Consts](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/const.py): shared config · Cross-cutting constants used across the package. · `src/ncku_course/const.py`
- [Utils](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/_inner_utils.py): shared helpers · Common helper functions shared by multiple modules. · `src/ncku_course/_inner_utils.py`

### NCKU access

Authentication, crawling, and captcha handling for the source site.

- [Auth](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/authenticate.py): session auth · Prepares access and session state for protected pages. · `src/ncku_course/authenticate.py`
- [Crawler](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/crawler.py): scraper · Fetches course data from the NCKU site, likely in parallel. · `src/ncku_course/crawler.py`
- [Captcha](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/captcha/main.py): ml solver · Captcha solving package and runtime model integration. · `src/ncku_course/captcha/main.py`
- [Predict](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/captcha/predict.py): inference · Model inference helpers for captcha recognition. · `src/ncku_course/captcha/predict.py`
- [Model](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/captcha/model_20240828_2152.h5): captcha model · Packaged trained model artifact used by the solver. · `src/ncku_course/captcha/model_20240828_2152.h5`

### Output delivery

Workbook generation and optional cloud/local artifact delivery.

- [Excel](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/excel.py): exporter · Transforms scraped records into a workbook artifact. · `src/ncku_course/excel.py`
- [OneDrive](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/api.py): cloud client · OneDrive integration boundary for uploads and storage. · `src/ncku_course/onedrive/api.py`
- [Upload](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/upload.py): delivery · Upload workflow for sending generated files to OneDrive. · `src/ncku_course/onedrive/upload.py`
- [Tokens](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/tokens.py): credentials · Token handling and credential persistence for OneDrive. · `src/ncku_course/onedrive/tokens.py`
- [Data](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/data.py): payloads · Shared data structures used by the OneDrive client. · `src/ncku_course/onedrive/data.py`
- [O-Utils](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/_internal_utils.py): client helpers · Internal helper functions for the OneDrive package. · `src/ncku_course/onedrive/_internal_utils.py`
- [Keys](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/key.py): secret helpers · Key-related handling for the OneDrive integration. · `src/ncku_course/onedrive/key.py`
- [Errors](https://github.com/heyihuang826/ncku_course/blob/HEAD/src/ncku_course/onedrive/exceptions.py): exceptions · OneDrive-specific exception types. · `src/ncku_course/onedrive/exceptions.py`

### External systems

Upstream and downstream services the pipeline depends on.

- [GitHub Actions](https://github.com/heyihuang826/ncku_course/blob/HEAD/.github/workflows/main.yml): scheduler · Scheduled CI runtime for unattended batch execution. · `.github/workflows/main.yml`
- **NCKU site**: upstream service · Primary source website for course data.
- **OneDrive cloud**: downstream service · Optional cloud destination for exported workbooks.

## Connections

- Main → Run flow: starts · Launcher invokes the main job flow.
- Run flow → Auth: prepares · Orchestration sets up access to the source site.
- Run flow → Crawler: collects · Job flow triggers course crawling.
- Run flow → Excel: exports · Job flow sends scraped data to workbook generation.
- Run flow → Upload: delivers · Job flow hands off finished artifacts for upload or storage.
- Auth → Captcha: handles captcha · Authentication relies on the captcha solver when verification is required.
- Captcha → Predict: runs inference · Solver uses inference helpers to interpret captcha images.
- Predict → Model: loads model · Inference depends on the packaged trained model.
- Crawler → NCKU site: requests data · Crawler talks to the NCKU course website.
- Auth → NCKU site: authenticates · Session setup interacts with the upstream site.
- Excel → Consts: uses · Excel export depends on shared constants.
- Excel → Utils: uses · Excel export reuses shared helpers.
- Upload → OneDrive: calls · Upload workflow uses the OneDrive client boundary.
- OneDrive → Tokens: manages · Client relies on token handling.
- OneDrive → Data: uses · Client uses shared data structures.
- OneDrive → O-Utils: uses · Client uses internal helper logic.
- OneDrive → Keys: uses · Client uses key handling helpers.
- OneDrive → Errors: raises · Client exposes OneDrive-specific failures.
- Upload → OneDrive cloud: uploads to · Delivery stage targets OneDrive when configured.
- GitHub Actions → Main: runs · Scheduled workflow launches the entrypoint in CI.
- GitHub Actions → Run flow: orchestrates · CI workflow executes the batch pipeline.
