# kshru9/web-crawler architecture

> Architecture diagram of the GitHub repository kshru9/web-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kshru9/web-crawler
- Repository: https://github.com/kshru9/web-crawler

## Overview

##### Project Type and Purpose
- This is a systems‐level, multi‐threaded web crawler and ranking toolkit.
- Written primarily in C++ (with Python scripts for PageRank and performance analysis).
- Core goals: crawl web pages concurrently, extract links/domains, compute simple and PageRank scores, and generate timing/analytics outputs.

##### High‐Level Architecture Components
A. Input Layer
- “INPUT/initialLinks.txt”: seed URLs for the crawler.
B. Core Crawler (C++ Executable)
1. Crawler Manager (main.cpp / Crawler.cpp/h)
- Reads seed URLs, enforces page/links limits & thread limits.
- Spawns worker threads or sleeps/wakes based on queue state.
2. Thread‐Safe Infrastructure (thread_safe/)
- thread_safe/queue_ts.cpp/h: URL frontier (FIFO).
- thread_safe/map_ts.cpp/h: visited pages & link graph.
- thread_safe/int_ts.cpp/h: atomic counters (visited count, working threads).
3. Worker Thread Modules
- Downloader (headers/downloaders.h + socket/OpenSSL): fetch HTML over HTTP/HTTPS.
- HTML Parser (headers/getLinks.h): regex‐based link extraction.
- Domain Extractor (headers/getDomain.h): parse host names.
- Ranking Updater (in C++): simple counter‐based rank on the fly.
4. Synchronization Strategy
- Two modes (build branches):
- Single Lock: one global mutex for shared state.
- Fine‐Grained Locks: per‐data‐structure locks / RW locks.
5. Output Writers
- OUTPUT/crawler_timings.csv: total crawl time vs threads.
- OUTPUT/th_timings.csv: per‐thread timing details.
- OUTPUT/pagerank.csv: raw link graph for Python PageRank.
C. Analytics & Ranking (Python Scripts)
- ranker.py: implements sampling‐based & iterative PageRank → updates pagerank.csv or prints to stdout.
- th_timings.py & test_bench.py: ingest CSVs and generate performance graphs using matplotlib.
D. Test Bench
- test_bench.cpp / test_bench.py: unit/integration tests for crawler performance and correctness.
E. Build & Deployment
- Makefile: targets for single‐threaded, multi‐threaded (single lock), multi‐threaded (thread-safe DS).
- Parameters: maxlinks, pagelimit, threads, rankerFlag.

##### Data Flows & Interactions
- Seed URLs → Crawler Manager → initial push to thread-safe queue.
- Crawler Manager monitors queue & working threads → spawns or sleeps.
- Worker Thread: pop URL → network fetch → parse HTML → extract links/domains → update shared state → write to CSV at end.
- Upon crawl completion: Python scripts read CSV outputs → compute PageRank or plot graphs.
- Test bench illustrates component correctness by driving the crawler and validating output.

##### Architectural Patterns & Design Principles
- Producer‐Consumer (Crawler Manager and Worker Threads) with thread-safe queues.
- Strategy Pattern for locking (single lock vs multiple locks).
- Modular layering: network I/O, parsing, data storage, ranking, analysis.
- Separation of concerns: C++ for crawling & basic ranking, Python for heavy analytics and graphing.

##### Essential Details to Capture
- Limits & parameters: maxlinks, pagelimit, threads, rankerFlag.
- Branches in repo structure (single-threaded, MT_singlelock, main).
- External dependencies: Pthread, OpenSSL, sockets, regex, Matplotlib.
- Test harnesses on both C++ and Python sides.

##### Final Checklist
- Every component is labeled.
- Extension points and build‐time decisions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    Input["initialLinks.txt"]:::data
    click Input "https://github.com/kshru9/web-crawler/blob/main/INPUT/initialLinks.txt"

    %% C++ Crawler Core
    subgraph "C++ Crawler Core"
        direction TB
        CrawlerMgr["Crawler Manager"]:::cppModule
        click CrawlerMgr "https://github.com/kshru9/web-crawler/blob/main/main.cpp"
        TSQueue["Thread-Safe Queue"]:::infra
        click TSQueue "https://github.com/kshru9/web-crawler/blob/main/thread_safe/queue_ts.cpp"
        TSMap["Thread-Safe Map"]:::infra
        click TSMap "https://github.com/kshru9/web-crawler/blob/main/thread_safe/map_ts.cpp"
        TSCounter["Thread-Safe Counters"]:::infra
        click TSCounter "https://github.com/kshru9/web-crawler/blob/main/thread_safe/int_ts.cpp"
        subgraph "Worker Modules"
            direction TB
            Downloader["Downloader"]:::cppModule
            click Downloader "https://github.com/kshru9/web-crawler/blob/main/headers/downloaders.h"
            Parser["HTML Parser"]:::cppModule
            click Parser "https://github.com/kshru9/web-crawler/blob/main/headers/getLinks.h"
            DomainExtractor["Domain Extractor"]:::cppModule
            click DomainExtractor "https://github.com/kshru9/web-crawler/blob/main/headers/getDomain.h"
            RankUpdater["Ranking Updater"]:::cppModule
            click RankUpdater "https://github.com/kshru9/web-crawler/blob/main/Crawler.cpp"
        end
        subgraph "Output Writers"
            direction TB
            CrawlTimeCSV["crawler_timings.csv"]:::data
            click CrawlTimeCSV "https://github.com/kshru9/web-crawler/blob/main/OUTPUT/crawler_timings.csv"
            ThreadTimeCSV["th_timings.csv"]:::data
            click ThreadTimeCSV "https://github.com/kshru9/web-crawler/blob/main/OUTPUT/th_timings.csv"
            PageRankCSV["pagerank.csv"]:::data
            click PageRankCSV "https://github.com/kshru9/web-crawler/blob/main/OUTPUT/pagerank.csv"
        end
        %% Synchronization modes annotation
        LockModes{"Locking Modes\n- Single Lock\n- Fine-Grained Locks"}:::infra
    end

    %% External Network
    Internet("Internet (HTTP/HTTPS)"):::network

    %% Python Analytics
    subgraph "Python Analytics"
        direction TB
        Ranker["PageRank Analyzer"]:::python
        click Ranker "https://github.com/kshru9/web-crawler/blob/main/ranker.py"
        TimingScript["Timing Analysis"]:::python
        click TimingScript "https://github.com/kshru9/web-crawler/blob/main/th_timings.py"
        PyTestBench["Test Bench (Python)"]:::python
        click PyTestBench "https://github.com/kshru9/web-crawler/blob/main/test_bench.py"
    end

    %% C++ Test Bench
    TestBenchCPP["Test Bench (C++)"]:::cppModule
    click TestBenchCPP "https://github.com/kshru9/web-crawler/blob/main/test_bench.cpp"

    %% Build & Deployment
    MakefileNode["Makefile"]:::config
    click MakefileNode "https://github.com/kshru9/web-crawler/tree/main/Makefile"
    Readme["README.md"]:::config
    click Readme "https://github.com/kshru9/web-crawler/blob/main/README.md"

    %% Data Flow
    Input -->|"seed URLs"| CrawlerMgr
    CrawlerMgr -->|"push URL"| TSQueue
    CrawlerMgr -->|"spawn/sleep threads"| LockModes
    TSQueue -->|"pop URL"| Downloader
    Downloader -->|"HTML page"| Parser
    Parser -->|"links/domains"| DomainExtractor
    DomainExtractor -->|"update"| TSMap
    Parser -->|"update rank"| RankUpdater
    TSMap -->|"visited records"| TSCounter
    TSCounter -->|"count metrics"| LockModes
    RankUpdater --> CrawlTimeCSV
    TSCounter --> ThreadTimeCSV
    TSMap --> PageRankCSV

    CrawlTimeCSV --> Ranker
    ThreadTimeCSV --> TimingScript
    PageRankCSV --> Ranker

    %% Network fetch
    Downloader --> Internet

    %% Testing
    TestBenchCPP --> CrawlerMgr
    PyTestBench --> Ranker

    %% Build interactions
    MakefileNode -.-> CrawlerMgr
    MakefileNode -.-> LockModes
    MakefileNode -.-> RankUpdater

    %% Styles
    classDef cppModule fill:#D0E6FF,stroke:#0366d6,stroke-width:2px
    classDef infra fill:#E8F1F8,stroke:#2F81F7,stroke-width:2px
    classDef data fill:#DFF5E0,stroke:#28A745,stroke-width:2px
    classDef network fill:#FFE5B4,stroke:#E36209,stroke-width:2px
    classDef python fill:#FFF4D6,stroke:#E6A700,stroke-width:2px
    classDef config fill:#F0F0F0,stroke:#6A737D,stroke-width:2px
```
