How heyihuang826/ncku_course works
This repository is a Python-based batch scraper and data export pipeline for NCKU course information.
High-level architecture
The project is organized as a small command-line/job runner with three major concerns:
1. Data acquisition from the NCKU course site
2. Captcha solving and authentication support
3. Post-processing into Excel and optional delivery to OneDrive or local storage
The system is designed to run either:
• manually on a local machine, or
• automatically through GitHub Actions on a schedule.
Main execution path
The runtime entry point appears to be:
• src/main.py as the top-level launcher
• src/ncku_course/run.py as the orchestration layer
The core job flow is:
1. Start the run.
2. Authenticate or establish the session needed for the NCKU site.
3. Crawl course data, likely using multithreaded requests.
4. Use the captcha model when the site requires verification.
5. Transform the collected data into an Excel workbook.
6. Save the file locally, upload it to OneDrive, or possibly persist it in the repository/GitHub Actions context.
Subsystems
1) Crawling subsystem
Located mainly in:
• src/ncku_course/crawler.py
This is the data collection boundary around the external NCKU website. Based on the README, it is responsible for:
• fetching course listings / course details
• handling repeated scraping at scale
• using multithreading to improve throughput
This subsystem is the main consumer of the site’s public or session-protected endpoints.
2) Authentication and captcha subsystem
Located in:
• src/ncku_course/authenticate.py
• src/ncku_course/captcha/
This is an important architectural boundary because the site can block scraping without verification.
The captcha package contains:
• captcha/main.py
• captcha/predict.py
• a trained model artifact (model_20240828_2152.h5 and SavedModel files)
This indicates the repository bundles a local ML-based captcha solver rather than relying on a third-party captcha service. Architecturally, this subsystem sits between the crawler and the website session flow:
• crawler hits protected pages
• captcha/authentication layer solves or bypasses the verification step
• session proceeds with authenticated requests
The model files are a deployment dependency, not just source code, so the captcha solver is part of the runtime environment.
3) Excel generation subsystem
Located in:
• src/ncku_course/excel.py
This subsystem converts scraped course data into a structured Excel file. It is the output formatting boundary for the whole application. The README’s example file suggests the output is a finished report workbook rather than raw CSV or JSON.
This layer likely depends on the data shape produced by the crawler and is responsible for organizing the final dataset into a user-consumable spreadsheet.
4) OneDrive integration subsystem
Located in:
• src/ncku_course/onedrive/
This is a self-contained integration boundary for cloud storage. It includes:
• API interaction code
• token handling
• key/credential logic
• upload helpers
• custom exceptions and shared utilities
From the repository structure, this looks like a dedicated client wrapper around OneDrive rather than ad hoc upload code. That separation matters architecturally because the core scraping pipeline does not need to know OneDrive protocol details; it only hands off the generated Excel file.
5) Shared utilities and configuration
Located in:
• src/ncku_course/_inner_utils.py
• src/ncku_course/const.py
• src/ncku_course/onedrive/_internal_utils.py
These files likely hold cross-cutting helpers, constants, and shared internal logic used by multiple subsystems.
They are not primary business logic boundaries, but they support:
• request/session handling
• file naming or path conventions
• reusable API helpers
• token/secret-related plumbing
Data flow
A typical run follows this path:
1. Start / orchestration
- main.py or run.py initializes the job and chooses the storage target.
2. Session preparation
- authentication logic prepares access to the course site.
- if captcha appears, the captcha model is used.
3. Scraping
- the crawler collects course data, likely in parallel threads.
4. Normalization
- the raw scraped records are shaped into a consistent in-memory representation.
5. Excel export
- the workbook is generated.
6. Delivery
- either:
- upload to OneDrive, or
- save locally
- in GitHub Actions mode, the workflow can also preserve the file in the repo-based automation flow, depending on configuration.
External systems and boundaries
NCKU course website
This is the primary upstream data source. The whole repository exists to interact with it reliably over time.
OneDrive
Optional downstream storage target. It is used as a cloud sink for the generated Excel file.
GitHub Actions
The repository includes .github/workflows/main.yml, which indicates scheduled automation. This turns the scraper into a periodic batch job running in CI rather than only a user-invoked tool.
The workflow boundary matters because the project is meant to run:
• without human interaction,
• on a schedule,
• with secrets/configuration injected by the CI environment.
Notable architectural characteristics
• Batch-oriented, not request/response
- This is a scheduled data pipeline, not a web server or interactive app.
• Anti-blocking emphasis
- Captcha-solving is a first-class subsystem, showing the scraper is designed to survive defensive website behavior.
• Separation of concerns
- crawling, captcha/auth, Excel output, and OneDrive upload are isolated into distinct packages/modules.
• Portable output
- The final artifact is an Excel file, making the repository useful both for automated distribution and manual inspection.
• Operational flexibility
- Supports local execution, GitHub Actions execution, and different storage destinations.
Technology/runtime implications
The project targets:
• Python 3.8.10 as stated in the README
• a machine-learning captcha model packaged with the repo
• GitHub Actions for scheduled automation
• OneDrive integration for optional cloud delivery
Suggested diagram boundaries
If you were drawing the architecture, the cleanest boundaries would be:
• Job entry / orchestration
- main.py, run.py
• NCKU site access
- authenticate.py, crawler.py, captcha package
• Data transformation
- excel.py
• Artifact delivery
- onedrive/ plus local file output
• External services
- NCKU website
- OneDrive
- GitHub Actions scheduler/runtime
In short, this repo is a scheduled scraping pipeline with ML-assisted captcha handling and flexible Excel output delivery.
Main entry points: api.py OneDrive · run.py Run flow · authenticate.py Auth · excel.py Excel
Components 20
Job runner
Entry points and scheduled execution for the batch pipeline.
- Main · Top-level launcher for local or CI execution.
src/main.py - Run flow · Coordinates the end-to-end batch job.
src/ncku_course/run.py - Consts · Cross-cutting constants used across the package.
src/ncku_course/const.py - Utils · Common helper functions shared by multiple modules.
src/ncku_course/_inner_utils.py
NCKU access
Authentication, crawling, and captcha handling for the source site.
- Auth · Prepares access and session state for protected pages.
src/ncku_course/authenticate.py - Crawler · Fetches course data from the NCKU site, likely in parallel.
src/ncku_course/crawler.py - Captcha · Captcha solving package and runtime model integration.
src/ncku_course/captcha/main.py - Predict · Model inference helpers for captcha recognition.
src/ncku_course/captcha/predict.py - Model · Packaged trained model artifact used by the solver.
src/ncku_course/captcha/model_20240828_2152.h5
Output delivery
Workbook generation and optional cloud/local artifact delivery.
- Excel · Transforms scraped records into a workbook artifact.
src/ncku_course/excel.py - OneDrive · OneDrive integration boundary for uploads and storage.
src/ncku_course/onedrive/api.py - Upload · Upload workflow for sending generated files to OneDrive.
src/ncku_course/onedrive/upload.py - Tokens · Token handling and credential persistence for OneDrive.
src/ncku_course/onedrive/tokens.py - Data · Shared data structures used by the OneDrive client.
src/ncku_course/onedrive/data.py - O-Utils · Internal helper functions for the OneDrive package.
src/ncku_course/onedrive/_internal_utils.py - Keys · Key-related handling for the OneDrive integration.
src/ncku_course/onedrive/key.py - Errors · OneDrive-specific exception types.
src/ncku_course/onedrive/exceptions.py
External systems
Upstream and downstream services the pipeline depends on.
- GitHub Actions · Scheduled CI runtime for unattended batch execution.
.github/workflows/main.yml - NCKU site · Primary source website for course data.
- OneDrive cloud · Optional cloud destination for exported workbooks.
Connections 21
- Main → Run flow: starts · Launcher invokes the main job flow.
- Run flow → Auth: prepares · Orchestration sets up access to the source site.
- Run flow → Crawler: collects · Job flow triggers course crawling.
- Run flow → Excel: exports · Job flow sends scraped data to workbook generation.
- Run flow → Upload: delivers · Job flow hands off finished artifacts for upload or storage.
- Auth → Captcha: handles captcha · Authentication relies on the captcha solver when verification is required.
- Captcha → Predict: runs inference · Solver uses inference helpers to interpret captcha images.
- Predict → Model: loads model · Inference depends on the packaged trained model.
- Crawler → NCKU site: requests data · Crawler talks to the NCKU course website.
- Auth → NCKU site: authenticates · Session setup interacts with the upstream site.
- Excel → Consts: uses · Excel export depends on shared constants.
- Excel → Utils: uses · Excel export reuses shared helpers.
- Upload → OneDrive: calls · Upload workflow uses the OneDrive client boundary.
- OneDrive → Tokens: manages · Client relies on token handling.
- OneDrive → Data: uses · Client uses shared data structures.
- OneDrive → O-Utils: uses · Client uses internal helper logic.
- OneDrive → Keys: uses · Client uses key handling helpers.
- OneDrive → Errors: raises · Client exposes OneDrive-specific failures.
- Upload → OneDrive cloud: uploads to · Delivery stage targets OneDrive when configured.
- GitHub Actions → Main: runs · Scheduled workflow launches the entrypoint in CI.
- GitHub Actions → Run flow: orchestrates · CI workflow executes the batch pipeline.