# nanmicoder/crawlertutorial architecture

> Architecture diagram of the GitHub repository nanmicoder/crawlertutorial, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nanmicoder/crawlertutorial
- Repository: https://github.com/nanmicoder/crawlertutorial

## Overview

##### Project type and purpose
- This is primarily a *static documentation/tutorial site* for web-scraping (爬虫) written in Chinese, with *accompanying Python example code*.
- It uses VitePress (Vue.js + Markdown) to generate the docs and GitHub Actions to build & deploy to GitHub Pages (or similar).
- Under the “源代码” directory are the step-by-step Python scripts illustrating introductory, intermediate, and advanced scraping techniques.

##### File structure
Top-level:
- .github/workflows/deploy.yml → CI/CD pipeline definition
- package.json / package-lock.json → Node/VitePress dependencies & build scripts
- README.md → project overview and links
- docs/ → VitePress site source (Markdown content, custom theme, static assets)
- 源代码/ → Python tutorial code, organized by tutorial section (入门, 进阶, 高级)
- 爬虫进阶, 爬虫高级 folders → Chinese-named readmes (mirrors “源代码/爬虫进阶” etc.)

##### README for additional insights
- It frames the repo as a *continuous-update tutorial*, links to the live site, and invites learners to clone and run code locally.

a. Main components:
1. GitHub Repo (source)
2. Developer (maintainer) environment
3. CI/CD (GitHub Actions)
4. VitePress build layer (Node.js, Vue.js)
5. Static site output (HTML/CSS/JS)
6. Hosting platform (GitHub Pages)
7. Learner/End-user (visiting site & cloning Python code)
8. Python runtime environment (for running example scripts)
b. Relationships & data flow:
- Developer commits → pushes to GitHub → triggers CI (deploy.yml)
- CI pulls Markdown & theme → runs `npm install` & `vitepress build` → generates static site
- CI deploys to hosting (GitHub Pages)
- Learner visits site in browser → reads docs → clones “源代码” → sets up Python venv → runs example scripts
- Python scripts interact with external websites (HTTP requests), parse data, optionally store locally (SQLite or remote DB)
c. Architectural patterns & principles:
- *Static Site Generation* (SSG) for documentation (decoupled from any backend)
- *Separation of concerns*:
- Docs front-end (Markdown + Vue theme)
- Example code (Python modules illustrating various patterns: synchronous vs async, storage abstraction, concurrency models)
- *CI/CD automation* for build/deploy
- *Modular tutorial code* showing progression (入门 → 进阶 → 高级)
d. Key technologies/frameworks:
- VitePress (Vue.js, Vite) for docs
- Node.js / npm for build tooling
- GitHub Actions for CI/CD
- Python 3.x for code examples
- Requests / aiohttp (implicit) for HTTP in examples
- SQLite or async DB adapters for storage demos

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Repository Layer
    subgraph "GitHub Repository"
        direction TB
        RepoRoot["Repo Root"]:::repo
        README["README.md"]:::repo
        PackageJSON["package.json"]:::repo
        PackageLock["package-lock.json"]:::repo
        subgraph ".github/workflows"
            direction TB
            CI_YML["deploy.yml"]:::ci
        end
        subgraph "docs/.vitepress"
            direction TB
            Config["config.mjs"]:::docs
            subgraph "theme"
                direction TB
                ThemeDir["theme/"]:::docs
                DynamicAds["DynamicAds.vue"]:::docs
                MyLayout["MyLayout.vue"]:::docs
                CustomCSS["custom.css"]:::docs
                ThemeIndex["index.js"]:::docs
            end
        end
        subgraph "docs Content"
            direction TB
            DocsIndex["docs/index.md"]:::docs
            DocsBeginner["docs/爬虫入门/"]:::docs
            DocsIntermediate["docs/爬虫进价/README.md"]:::docs
            DocsAdvanced["docs/高级爬虫/README.md"]:::docs
        end
        subgraph "docs Static Assets"
            direction TB
            StaticAssets["docs/static/"]:::docs
        end
    end

    %% CI/CD Layer
    CI["GitHub Actions CI/CD"]:::ci

    %% Build Layer
    subgraph "Build & Generate Static Site"
        direction TB
        NodeTool["Node.js & npm"]:::docs
        VitePressBuild["VitePress Build"]:::docs
        SiteOutput["Static Site (HTML/CSS/JS)"]:::docs
    end

    %% Hosting Layer
    Hosting["GitHub Pages"]:::external

    %% Browser Layer
    Browser["Learner Browser"]:::external

    %% Python Tutorial Environment
    subgraph "Python Tutorial Environment"
        direction TB
        PythonEnv["Python Env"]:::python
        subgraph "Tutorial Code"
            direction TB
            PyBeginner["入门 Scripts"]:::python
            PyIntermediate["进阶 main.py"]:::python
            PyAdvanced["高级 main.py"]:::python
        end
    end

    %% External Target Websites
    Target["Target Websites"]:::external

    %% Connections
    Developer["Developer"]:::external
    Developer -->|"git push"| RepoRoot
    RepoRoot -->|triggers| CI
    CI_YML --> CI
    CI -->|"npm install & vitepress build"| VitePressBuild
    VitePressBuild -->|generates| SiteOutput
    SiteOutput -->|deploy| Hosting
    Hosting -->|serves| Browser
    Browser -->|reads docs| SiteOutput
    Browser -->|clone src| RepoRoot
    RepoRoot -->|git clone| PythonEnv
    PythonEnv -->|runs| PyBeginner
    PythonEnv -->|runs| PyIntermediate
    PythonEnv -->|runs| PyAdvanced
    PyBeginner -->|HTTP GET| Target
    PyIntermediate -->|HTTP GET| Target
    PyAdvanced -->|HTTP GET| Target

    %% Click Events
    click CI_YML "https://github.com/nanmicoder/crawlertutorial/blob/main/.github/workflows/deploy.yml"
    click README "https://github.com/nanmicoder/crawlertutorial/blob/main/README.md"
    click PackageJSON "https://github.com/nanmicoder/crawlertutorial/blob/main/package.json"
    click PackageLock "https://github.com/nanmicoder/crawlertutorial/blob/main/package-lock.json"
    click Config "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/.vitepress/config.mjs"
    click ThemeDir "https://github.com/nanmicoder/crawlertutorial/tree/main/docs/.vitepress/theme/"
    click DynamicAds "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/.vitepress/theme/DynamicAds.vue"
    click MyLayout "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/.vitepress/theme/MyLayout.vue"
    click CustomCSS "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/.vitepress/theme/custom.css"
    click ThemeIndex "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/.vitepress/theme/index.js"
    click DocsIndex "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/index.md"
    click DocsBeginner "https://github.com/nanmicoder/crawlertutorial/tree/main/docs/爬虫入门/"
    click DocsIntermediate "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/爬虫进价/README.md"
    click DocsAdvanced "https://github.com/nanmicoder/crawlertutorial/blob/main/docs/高级爬虫/README.md"
    click StaticAssets "https://github.com/nanmicoder/crawlertutorial/tree/main/docs/static/"
    click PyBeginner "https://github.com/nanmicoder/crawlertutorial/tree/main/源代码/爬虫入门/"
    click PyIntermediate "https://github.com/nanmicoder/crawlertutorial/blob/main/源代码/爬虫进阶/main.py"
    click PyAdvanced "https://github.com/nanmicoder/crawlertutorial/blob/main/源代码/高级爬虫/main.py"

    %% Styles
    classDef docs fill:#D0E8FF,stroke:#007ACC,color:#000
    classDef repo fill:#E8F5FF,stroke:#005A9C,color:#000
    classDef ci fill:#DFF8E0,stroke:#28A745,color:#000
    classDef python fill:#FFE8D0,stroke:#DA8C10,color:#000
    classDef external fill:#F0F0F0,stroke:#888888,color:#000
```
