How nanmicoder/mediacrawler works
MediaCrawler is a multi-platform public-media crawler with both a command-line entry point and a WebUI. The main workflow selects a platform and crawl mode, authenticates through a browser session, fetches search, post, comment, or creator data, and saves results; optional media downloading and proxy rotation support collection. The README also documents a separate API/WebUI workflow for configuring runs, viewing live status and logs, and previewing or exporting data. Some crawler, API, and storage internals were not sampled, so uncertain implementation links are omitted or shown as unverified.
Main entry points: main.py CLI entry · xhs Xiaohongshu · App.tsx WebUI · douyin Douyin
Components 26
Entry and control
- CLI entry
main.py - Run options
cmd_arg/arg.py - Crawler contract
base/base_crawler.py
Platform collection
Collection support
- Browser login
base/base_crawler.py - Proxy rotation
proxy/proxy_ip_pool.py - Proxy client mixin
proxy/proxy_mixin.py - Media download
media_downloader/downloader.py
Data outputs
- Excel output
store/excel_store_base.py - Database output
database/db.py
Web interface
- WebUI
webui/src/App.tsx - API service
api/main.py - Crawler API
api/routers/crawler.py - Crawler manager
api/services/crawler_manager.py - Data API
api/routers/data.py - Live status
api/routers/websocket.py - Data preview
webui/src/components/data/preview/DataPreviewTable.tsx
Other components
- Operator
- Social platforms
- Web operator
Connections 35
- Operator → CLI entry: runs
README.md - CLI entry → Run options: parses options
main.py - CLI entry → Crawler contract: creates crawler
main.py - CLI entry → Xiaohongshu: dispatches platform
main.py - CLI entry → Douyin: dispatches platform
main.py - CLI entry → Kuaishou: dispatches platform
main.py - CLI entry → Bilibili: dispatches platform
main.py - CLI entry → Weibo: dispatches platform
main.py - CLI entry → Tieba: dispatches platform
main.py - CLI entry → Zhihu: dispatches platform
main.py - Xiaohongshu → Social platforms: requests public data
README.md - Douyin → Social platforms: requests public data
README.md - Kuaishou → Social platforms: requests public data
README.md - Bilibili → Social platforms: requests public data
README.md - Weibo → Social platforms: requests public data
README.md - Tieba → Social platforms: requests public data
README.md - Zhihu → Social platforms: requests public data
README.md - Xiaohongshu → Browser login: uses browser session
media_platform/xhs/client.py - Douyin → Browser login: uses browser session
media_platform/douyin/client.py - Xiaohongshu → Proxy client mixin: uses proxy support
media_platform/xhs/client.py - Douyin → Proxy client mixin: uses proxy support
media_platform/douyin/client.py - Proxy client mixin → Proxy rotation: refreshes from
proxy/proxy_mixin.py - Xiaohongshu → Media download: downloads optional media
- CLI entry → Excel output: flushes output
main.py - CLI entry → Database output: initializes database
main.py - Operator → Web operator: uses separately
- Web operator → WebUI: configures and views
README.md - WebUI → API service: calls API
README.md - API service → Crawler API: mounts routes
- Crawler API → Crawler manager: manages runs
- API service → Data API: mounts routes
- API service → Live status: mounts routes
- WebUI → Data preview: previews results
README.md - WebUI → Live status: receives live status
README.md - WebUI → Data API: loads and exports data
README.md