# nanmicoder/mediacrawler architecture

> Architecture diagram of the GitHub repository nanmicoder/mediacrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nanmicoder/mediacrawler
- Repository: https://github.com/nanmicoder/mediacrawler

## Overview

MediaCrawler is a multi-platform public-media crawler with both a command-line entry point and a WebUI. The main workflow selects a platform and crawl mode, authenticates through a browser session, fetches search, post, comment, or creator data, and saves results; optional media downloading and proxy rotation support collection. The README also documents a separate API/WebUI workflow for configuring runs, viewing live status and logs, and previewing or exporting data. Some crawler, API, and storage internals were not sampled, so uncertain implementation links are omitted or shown as unverified.

Main entry points: [`main.py`](https://github.com/nanmicoder/mediacrawler/blob/main/main.py) (CLI entry), [`media_platform/xhs`](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/xhs) (Xiaohongshu), [`webui/src/App.tsx`](https://github.com/nanmicoder/mediacrawler/blob/main/webui/src/App.tsx) (WebUI), [`media_platform/douyin`](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/douyin) (Douyin)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and control"]
  node_cli["CLI entry<br/>[main.py]"]
  node_arguments["Run options<br/>[arg.py]"]
  node_crawler_base["Crawler contract<br/>[base_crawler.py]"]
end

subgraph group_platforms["Platform collection"]
  node_xhs["Xiaohongshu"]
  node_douyin["Douyin"]
  node_kuaishou["Kuaishou"]
  node_bilibili["Bilibili"]
  node_weibo["Weibo"]
  node_tieba["Tieba"]
  node_zhihu["Zhihu"]
end

subgraph group_runtime["Collection support"]
  node_browser_auth["Browser login<br/>[base_crawler.py]"]
  node_proxy_pool["Proxy rotation<br/>[proxy_ip_pool.py]"]
  node_proxy_mixin["Proxy client mixin<br/>[proxy_mixin.py]"]
  node_media_download["Media download<br/>[downloader.py]"]
end

subgraph group_persistence["Data outputs"]
  node_excel_store["Excel output"]
  node_database[("Database output<br/>[db.py]")]
end

subgraph group_web["Web interface"]
  node_webui["WebUI<br/>[App.tsx]"]
  node_web_api["API service<br/>[main.py]"]
  node_crawler_api["Crawler API<br/>[crawler.py]"]
  node_crawler_manager["Crawler manager<br/>[crawler_manager.py]"]
  node_data_api["Data API<br/>[data.py]"]
  node_websocket_api["Live status<br/>[websocket.py]"]
  node_web_data["Data preview"]
end

node_user(("Operator"))
node_social_platforms(("Social platforms"))
node_web_user(("Web operator"))

node_user -->|"runs"| node_cli
node_cli -->|"parses options"| node_arguments
node_cli -->|"creates crawler"| node_crawler_base
node_cli -->|"dispatches platform"| node_xhs
node_cli -->|"dispatches platform"| node_douyin
node_cli -->|"dispatches platform"| node_kuaishou
node_cli -->|"dispatches platform"| node_bilibili
node_cli -->|"dispatches platform"| node_weibo
node_cli -->|"dispatches platform"| node_tieba
node_cli -->|"dispatches platform"| node_zhihu
node_xhs -->|"requests public data"| node_social_platforms
node_douyin -->|"requests public data"| node_social_platforms
node_kuaishou -->|"requests public data"| node_social_platforms
node_bilibili -->|"requests public data"| node_social_platforms
node_weibo -->|"requests public data"| node_social_platforms
node_tieba -->|"requests public data"| node_social_platforms
node_zhihu -->|"requests public data"| node_social_platforms
node_xhs -.->|"uses browser session"| node_browser_auth
node_douyin -.->|"uses browser session"| node_browser_auth
node_xhs -->|"uses proxy support"| node_proxy_mixin
node_douyin -->|"uses proxy support"| node_proxy_mixin
node_proxy_mixin -->|"refreshes from"| node_proxy_pool
node_xhs -.->|"downloads optional media"| node_media_download
node_cli -->|"flushes output"| node_excel_store
node_cli -->|"initializes database"| node_database
node_user -.->|"uses separately"| node_web_user
node_web_user -->|"configures and views"| node_webui
node_webui -->|"calls API"| node_web_api
node_web_api -->|"mounts routes"| node_crawler_api
node_crawler_api -.->|"manages runs"| node_crawler_manager
node_web_api -->|"mounts routes"| node_data_api
node_web_api -->|"mounts routes"| node_websocket_api
node_webui -->|"previews results"| node_web_data
node_webui -.->|"receives live status"| node_websocket_api
node_webui -.->|"loads and exports data"| node_data_api

click node_cli "https://github.com/nanmicoder/mediacrawler/blob/main/main.py"
click node_arguments "https://github.com/nanmicoder/mediacrawler/blob/main/cmd_arg/arg.py"
click node_crawler_base "https://github.com/nanmicoder/mediacrawler/blob/main/base/base_crawler.py"
click node_xhs "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/xhs"
click node_douyin "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/douyin"
click node_kuaishou "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/kuaishou"
click node_bilibili "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/bilibili"
click node_weibo "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/weibo"
click node_tieba "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/tieba"
click node_zhihu "https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/zhihu"
click node_browser_auth "https://github.com/nanmicoder/mediacrawler/blob/main/base/base_crawler.py"
click node_proxy_pool "https://github.com/nanmicoder/mediacrawler/blob/main/proxy/proxy_ip_pool.py"
click node_proxy_mixin "https://github.com/nanmicoder/mediacrawler/blob/main/proxy/proxy_mixin.py"
click node_media_download "https://github.com/nanmicoder/mediacrawler/blob/main/media_downloader/downloader.py"
click node_excel_store "https://github.com/nanmicoder/mediacrawler/blob/main/store/excel_store_base.py"
click node_database "https://github.com/nanmicoder/mediacrawler/blob/main/database/db.py"
click node_webui "https://github.com/nanmicoder/mediacrawler/blob/main/webui/src/App.tsx"
click node_web_api "https://github.com/nanmicoder/mediacrawler/blob/main/api/main.py"
click node_crawler_api "https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/crawler.py"
click node_crawler_manager "https://github.com/nanmicoder/mediacrawler/blob/main/api/services/crawler_manager.py"
click node_data_api "https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/data.py"
click node_websocket_api "https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/websocket.py"
click node_web_data "https://github.com/nanmicoder/mediacrawler/blob/main/webui/src/components/data/preview/DataPreviewTable.tsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_arguments,node_crawler_base toneBlue
class node_xhs,node_douyin,node_kuaishou,node_bilibili,node_weibo,node_tieba,node_zhihu toneAmber
class node_browser_auth,node_proxy_pool,node_proxy_mixin,node_media_download toneMint
class node_excel_store,node_database toneRose
class node_webui,node_web_api,node_crawler_api,node_crawler_manager,node_data_api,node_websocket_api,node_web_data,node_user,node_social_platforms,node_web_user toneIndigo
```

## Components

### Entry and control

- [CLI entry](https://github.com/nanmicoder/mediacrawler/blob/main/main.py): `main.py`
- [Run options](https://github.com/nanmicoder/mediacrawler/blob/main/cmd_arg/arg.py): `cmd_arg/arg.py`
- [Crawler contract](https://github.com/nanmicoder/mediacrawler/blob/main/base/base_crawler.py): `base/base_crawler.py`

### Platform collection

- [Xiaohongshu](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/xhs): `media_platform/xhs`
- [Douyin](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/douyin): `media_platform/douyin`
- [Kuaishou](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/kuaishou): `media_platform/kuaishou`
- [Bilibili](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/bilibili): `media_platform/bilibili`
- [Weibo](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/weibo): `media_platform/weibo`
- [Tieba](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/tieba): `media_platform/tieba`
- [Zhihu](https://github.com/nanmicoder/mediacrawler/tree/main/media_platform/zhihu): `media_platform/zhihu`

### Collection support

- [Browser login](https://github.com/nanmicoder/mediacrawler/blob/main/base/base_crawler.py): `base/base_crawler.py`
- [Proxy rotation](https://github.com/nanmicoder/mediacrawler/blob/main/proxy/proxy_ip_pool.py): `proxy/proxy_ip_pool.py`
- [Proxy client mixin](https://github.com/nanmicoder/mediacrawler/blob/main/proxy/proxy_mixin.py): `proxy/proxy_mixin.py`
- [Media download](https://github.com/nanmicoder/mediacrawler/blob/main/media_downloader/downloader.py): `media_downloader/downloader.py`

### Data outputs

- [Excel output](https://github.com/nanmicoder/mediacrawler/blob/main/store/excel_store_base.py): `store/excel_store_base.py`
- [Database output](https://github.com/nanmicoder/mediacrawler/blob/main/database/db.py): `database/db.py`

### Web interface

- [WebUI](https://github.com/nanmicoder/mediacrawler/blob/main/webui/src/App.tsx): `webui/src/App.tsx`
- [API service](https://github.com/nanmicoder/mediacrawler/blob/main/api/main.py): `api/main.py`
- [Crawler API](https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/crawler.py): `api/routers/crawler.py`
- [Crawler manager](https://github.com/nanmicoder/mediacrawler/blob/main/api/services/crawler_manager.py): `api/services/crawler_manager.py`
- [Data API](https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/data.py): `api/routers/data.py`
- [Live status](https://github.com/nanmicoder/mediacrawler/blob/main/api/routers/websocket.py): `api/routers/websocket.py`
- [Data preview](https://github.com/nanmicoder/mediacrawler/blob/main/webui/src/components/data/preview/DataPreviewTable.tsx): `webui/src/components/data/preview/DataPreviewTable.tsx`

### Other components

- **Operator**
- **Social platforms**
- **Web operator**

## Connections

- Operator → CLI entry: runs (evidence: `README.md`)
- CLI entry → Run options: parses options (evidence: `main.py`)
- CLI entry → Crawler contract: creates crawler (evidence: `main.py`)
- CLI entry → Xiaohongshu: dispatches platform (evidence: `main.py`)
- CLI entry → Douyin: dispatches platform (evidence: `main.py`)
- CLI entry → Kuaishou: dispatches platform (evidence: `main.py`)
- CLI entry → Bilibili: dispatches platform (evidence: `main.py`)
- CLI entry → Weibo: dispatches platform (evidence: `main.py`)
- CLI entry → Tieba: dispatches platform (evidence: `main.py`)
- CLI entry → Zhihu: dispatches platform (evidence: `main.py`)
- Xiaohongshu → Social platforms: requests public data (evidence: `README.md`)
- Douyin → Social platforms: requests public data (evidence: `README.md`)
- Kuaishou → Social platforms: requests public data (evidence: `README.md`)
- Bilibili → Social platforms: requests public data (evidence: `README.md`)
- Weibo → Social platforms: requests public data (evidence: `README.md`)
- Tieba → Social platforms: requests public data (evidence: `README.md`)
- Zhihu → Social platforms: requests public data (evidence: `README.md`)
- Xiaohongshu → Browser login: uses browser session (evidence: `media_platform/xhs/client.py`)
- Douyin → Browser login: uses browser session (evidence: `media_platform/douyin/client.py`)
- Xiaohongshu → Proxy client mixin: uses proxy support (evidence: `media_platform/xhs/client.py`)
- Douyin → Proxy client mixin: uses proxy support (evidence: `media_platform/douyin/client.py`)
- Proxy client mixin → Proxy rotation: refreshes from (evidence: `proxy/proxy_mixin.py`)
- Xiaohongshu → Media download: downloads optional media
- CLI entry → Excel output: flushes output (evidence: `main.py`)
- CLI entry → Database output: initializes database (evidence: `main.py`)
- Operator → Web operator: uses separately
- Web operator → WebUI: configures and views (evidence: `README.md`)
- WebUI → API service: calls API (evidence: `README.md`)
- API service → Crawler API: mounts routes
- Crawler API → Crawler manager: manages runs
- API service → Data API: mounts routes
- API service → Live status: mounts routes
- WebUI → Data preview: previews results (evidence: `README.md`)
- WebUI → Live status: receives live status (evidence: `README.md`)
- WebUI → Data API: loads and exports data (evidence: `README.md`)
