How keyvanarasteh/darkwebcrawler works
Project type and purpose
• This is a single‑node, multi‑threaded, Python‑based crawler for dark‑web domains.
• It’s essentially a producer‑consumer system: seeds → queue → worker threads → database → reporting.
File structure
• Top‑level Python modules:
• main.py (orchestrator / entry point)
• crawler.py (crawl logic per page)
• crawler_proc.py (possibly subprocess or helper for crawling)
• database.py (DB connection & CRUD)
• connection.py (Tor / HTTP connections with user‑agents)
• pub_methods.py (shared utilities: filtering, regex, parsing)
• reporting.py (post‑crawl reporting & exports)
• settings.py (configuration loader)
• Data folders:
• assets/user-agents.dat (rotating UA list)
• temp/filters.dat (keyword filters)
• queue/x.txt (disk‑backed queue?)
• db_tables/*.sql (schema & seed scripts)
• log/log.txt (application logs)
• Concurrency layout:
• threads/thread.txt & threads_sub/… imply thread management or logging per thread.
Extract architecture hints from README
• Multi‑thread high‑performance crawler
• Keyword & domain filtering (simple + regex)
• MySQL backend
Main components
a. Orchestrator (main.py)
b. Configuration Loader (settings.py + assets/)
c. Queue Manager (in‑memory or disk via queue/x.txt)
d. Thread Pool Manager (spawned by main.py)
e. Crawler Workers (crawler.py + crawler_proc.py + connection.py)
f. Filtering Module (pub_methods.py + temp/filters.dat)
g. Database Layer (database.py → MySQL)
h. Reporting Module (reporting.py)
i. Logger (log/log.txt)
Relationships & interactions
• On startup, Orchestrator loads Configuration.
• Seeds initial URLs/domains into Queue Manager.
• Thread Pool Manager polls Queue for new tasks.
• Each Crawler Worker:
1. Grabs next URL/domain from Queue.
2. Uses Connection component to fetch page over Tor (user‑agents.dat).
3. Parses content, extracts links.
4. Sends extracted links through Filtering Module.
5. Accepted links re‑enqueue into Queue.
6. Results (metadata, checksums, filters applied) sent to Database Layer.
• Reporting Module aggregates data from MySQL for analytics/exports.
• Logger receives events across all components.
Design patterns & principles
• Producer‑Consumer (Queue + Thread Pool)
• Layered Architecture (config → orchestration → workers → persistence → reporting)
• Single Responsibility (each module has a clear purpose)
• Configuration‑Driven (settings.py + assets/dat files)
Technologies & frameworks to annotate
• Python 3.x
• MySQL (schema in db_tables/)
• Tor or socks proxy via connection.py
• Multi‑threading (threading or multiprocessing in crawler_proc.py)
• Regex and simple string filters
Final touches for clarity
• Every component has a clear, unique label.
• Optionally, include the sequence: Initialize → Crawl Loop → Terminate.