How arifshaon/webcrawlmanager works
SWM is a local web-archive manager: a curator creates a job through the dashboard or CLI, a worker runs a browser-based capture, and SWM preserves the resulting WARC and descriptive records in collections for later replay and search. Social-network and YouTube captures add platform-specific extraction and media preservation. The README documents quality checks and indexing; their detailed runtime wiring is not shown in the excerpts, so those links are omitted or marked unverified.
Main entry points: worker.py Job worker · facebook.py Social captures · redaction.py Secret redaction · collections.py Collections
Components 24
Job management
- Web dashboard
webarc/server.py - Command line
webarc/cli.py - Job worker
webarc/worker.py - Collections
webarc/collections.py - Capture settings
webarc/config.py
Capture engines
- Automated crawl
webarc/crawler.py - Session recording
webarc/recorder.py - Browser capture
webarc/browser.py - Social captures
webarc/facebook.py - Instagram browser
webarc/instagram_browser.py - YouTube capture
webarc/youtube.py - X capture
webarc/x.py - Instagram browser engine
webarc/instagram_browser.py
Preservation records
- WARC archives
webarc/capture.py - Secret redaction
webarc/redaction.py - Descriptive metadata
webarc/metadata.py - Capture manifest
webarc/capture_manifest.py - Settings provenance
webarc/capture_settings.py - Cross-job deduplication
webarc/dedup_index.py
Archive access
- Local replay
webarc/replay.py - Search indexing
webarc/indexer.py - WARC indexer
warc-indexer/src/main/java/uk/bl/wa/indexer/WARCIndexer.java - Quality checks
webarc/changes.py
Other components
- Curator
Connections 28
- Curator → Web dashboard: uses
README.md - Curator → Command line: runs
README.md - Web dashboard → Job worker: submits jobs
- Command line → Job worker: starts jobs
- Job worker → Automated crawl: runs
webarc/worker.py - Job worker → Session recording: runs
webarc/worker.py - Job worker → Social captures: runs
webarc/worker.py - Job worker → Instagram browser: runs
webarc/worker.py - Job worker → YouTube capture: runs
webarc/worker.py - Job worker → X capture: runs
webarc/worker.py - Job worker → Collections: uses
webarc/worker.py - Automated crawl → Browser capture: uses
- Session recording → Browser capture: uses
- Instagram browser → Instagram browser engine: uses
webarc/instagram.py - Social captures → Secret redaction: redacts data
webarc/facebook.py - Instagram browser → Secret redaction: redacts data
webarc/instagram.py - X capture → Secret redaction: redacts data
webarc/x.py - YouTube capture → Secret redaction: redacts data
webarc/youtube.py - Collections → Cross-job deduplication: uses index
webarc/collections.py - Collections → Descriptive metadata: reads fields
webarc/collections.py - Social captures → Capture manifest: records capture
webarc/facebook.py - Social captures → WARC archives: writes archive
webarc/facebook.py - Descriptive metadata → WARC archives: embedded in
webarc/capture.py - Settings provenance → WARC archives: embedded in
webarc/capture_settings.py - Curator → Local replay: browses locally
README.md - Curator → Search indexing: requests search
README.md - Search indexing → WARC indexer: uses
- Curator → Quality checks: reviews
README.md