How kshru9/web-crawler works
Project Type and Purpose
• This is a systems‐level, multi‐threaded web crawler and ranking toolkit.
• Written primarily in C++ (with Python scripts for PageRank and performance analysis).
• Core goals: crawl web pages concurrently, extract links/domains, compute simple and PageRank scores, and generate timing/analytics outputs.
High‐Level Architecture Components
A. Input Layer
• “INPUT/initialLinks.txt”: seed URLs for the crawler.
B. Core Crawler (C++ Executable)
1. Crawler Manager (main.cpp / Crawler.cpp/h)
• Reads seed URLs, enforces page/links limits & thread limits.
• Spawns worker threads or sleeps/wakes based on queue state.
2. Thread‐Safe Infrastructure (thread_safe/)
• thread_safe/queue_ts.cpp/h: URL frontier (FIFO).
• thread_safe/map_ts.cpp/h: visited pages & link graph.
• thread_safe/int_ts.cpp/h: atomic counters (visited count, working threads).
3. Worker Thread Modules
• Downloader (headers/downloaders.h + socket/OpenSSL): fetch HTML over HTTP/HTTPS.
• HTML Parser (headers/getLinks.h): regex‐based link extraction.
• Domain Extractor (headers/getDomain.h): parse host names.
• Ranking Updater (in C++): simple counter‐based rank on the fly.
4. Synchronization Strategy
• Two modes (build branches):
• Single Lock: one global mutex for shared state.
• Fine‐Grained Locks: per‐data‐structure locks / RW locks.
5. Output Writers
• OUTPUT/crawler_timings.csv: total crawl time vs threads.
• OUTPUT/th_timings.csv: per‐thread timing details.
• OUTPUT/pagerank.csv: raw link graph for Python PageRank.
C. Analytics & Ranking (Python Scripts)
• ranker.py: implements sampling‐based & iterative PageRank → updates pagerank.csv or prints to stdout.
• th_timings.py & test_bench.py: ingest CSVs and generate performance graphs using matplotlib.
D. Test Bench
• test_bench.cpp / test_bench.py: unit/integration tests for crawler performance and correctness.
E. Build & Deployment
• Makefile: targets for single‐threaded, multi‐threaded (single lock), multi‐threaded (thread-safe DS).
• Parameters: maxlinks, pagelimit, threads, rankerFlag.
Data Flows & Interactions
• Seed URLs → Crawler Manager → initial push to thread-safe queue.
• Crawler Manager monitors queue & working threads → spawns or sleeps.
• Worker Thread: pop URL → network fetch → parse HTML → extract links/domains → update shared state → write to CSV at end.
• Upon crawl completion: Python scripts read CSV outputs → compute PageRank or plot graphs.
• Test bench illustrates component correctness by driving the crawler and validating output.
Architectural Patterns & Design Principles
• Producer‐Consumer (Crawler Manager and Worker Threads) with thread-safe queues.
• Strategy Pattern for locking (single lock vs multiple locks).
• Modular layering: network I/O, parsing, data storage, ranking, analysis.
• Separation of concerns: C++ for crawling & basic ranking, Python for heavy analytics and graphing.
Essential Details to Capture
• Limits & parameters: maxlinks, pagelimit, threads, rankerFlag.
• Branches in repo structure (single-threaded, MT_singlelock, main).
• External dependencies: Pthread, OpenSSL, sockets, regex, Matplotlib.
• Test harnesses on both C++ and Python sides.
Final Checklist
• Every component is labeled.
• Extension points and build‐time decisions.