How nanmicoder/crawlertutorial works
Project type and purpose
• This is primarily a *static documentation/tutorial site* for web-scraping (爬虫) written in Chinese, with *accompanying Python example code*.
• It uses VitePress (Vue.js + Markdown) to generate the docs and GitHub Actions to build & deploy to GitHub Pages (or similar).
• Under the “源代码” directory are the step-by-step Python scripts illustrating introductory, intermediate, and advanced scraping techniques.
File structure
Top-level:
• .github/workflows/deploy.yml → CI/CD pipeline definition
• package.json / package-lock.json → Node/VitePress dependencies & build scripts
• README.md → project overview and links
• docs/ → VitePress site source (Markdown content, custom theme, static assets)
• 源代码/ → Python tutorial code, organized by tutorial section (入门, 进阶, 高级)
• 爬虫进阶, 爬虫高级 folders → Chinese-named readmes (mirrors “源代码/爬虫进阶” etc.)
README for additional insights
• It frames the repo as a *continuous-update tutorial*, links to the live site, and invites learners to clone and run code locally.
a. Main components:
1. GitHub Repo (source)
2. Developer (maintainer) environment
3. CI/CD (GitHub Actions)
4. VitePress build layer (Node.js, Vue.js)
5. Static site output (HTML/CSS/JS)
6. Hosting platform (GitHub Pages)
7. Learner/End-user (visiting site & cloning Python code)
8. Python runtime environment (for running example scripts)
b. Relationships & data flow:
• Developer commits → pushes to GitHub → triggers CI (deploy.yml)
• CI pulls Markdown & theme → runs npm install & vitepress build → generates static site
• CI deploys to hosting (GitHub Pages)
• Learner visits site in browser → reads docs → clones “源代码” → sets up Python venv → runs example scripts
• Python scripts interact with external websites (HTTP requests), parse data, optionally store locally (SQLite or remote DB)
c. Architectural patterns & principles:
• *Static Site Generation* (SSG) for documentation (decoupled from any backend)
• *Separation of concerns*:
• Docs front-end (Markdown + Vue theme)
• Example code (Python modules illustrating various patterns: synchronous vs async, storage abstraction, concurrency models)
• *CI/CD automation* for build/deploy
• *Modular tutorial code* showing progression (入门 → 进阶 → 高级)
d. Key technologies/frameworks:
• VitePress (Vue.js, Vite) for docs
• Node.js / npm for build tooling
• GitHub Actions for CI/CD
• Python 3.x for code examples
• Requests / aiohttp (implicit) for HTTP in examples
• SQLite or async DB adapters for storage demos