# akhil298/web-crawler architecture

> Architecture diagram of the GitHub repository akhil298/web-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akhil298/web-crawler
- Repository: https://github.com/akhil298/web-crawler

## Overview

##### Project Type and Purpose
- This is a web scraping tool designed specifically for crawling multiple e-commerce domains to discover product URLs.
- It is built to scale, using parallel processing and modular design. The README details that it operates by scraping homepages, filtering URLs with regex, processing dynamic content (via scrolling), and then saving the found URLs to JSON files along with logging.

##### File Structure
- Key Directories and Files:
- main.py: The entry point that initiates and orchestrates the entire process.
- config/: Contains configuration parameters (in config.py), which determine behavior such as browser settings, scrolling parameters, and worker counts.
- scraping/: Contains the web scraping logic (using Selenium and BeautifulSoup) responsible for navigating pages and collecting raw URLs.
- processing/: Handles the extraction and refining of URLs through regex filtering and cleaning (processor.py).
- saving/: Manages the storage of processed data (saver.py) into JSON files.
- utils/: Provides shared utility functions that support URL processing and other common tasks.
- Test/sample pages for various e-commerce websites are present in directories like nykaafashion.com, www.tatacliq.com, www.virgio.com, and www.westside.com, which serve as both test input and reference HTML examples.
- Supporting files: product_links.json, endpoint.json, status_log.csv, and initial_status_log.csv for data storage and logging.

##### Readme Insights and Architectural Patterns
- The README describes a modular and scalable design focused on the stages of URL discovery, filtering, dynamic content handling, and data persistence.
- The crawler makes extensive use of parallel processing and configurable settings.
- Architectural emphasis should be given to distinct responsibilities: scraping, processing, and data saving. The separation of concerns hints at a layered approach where each module focuses on a specific part of the workflow.

a. Main Components to Include:
- Entry Point (main.py)
- Configuration Module (config/config.py) – Provide parameters like SHOW_BROWSER, MAX_SCROLLS, MAX_WORKERS, etc.
- Scraper Module (scraping/scraper.py) – Handles dynamic page interaction and extraction of potential URLs.
- Processing Module (processing/processor.py) – Applies regex filtering and cleans the extracted URLs.
- Saver Module (saving/saver.py) – Persists the final results to JSON files.
- Utility Functions (utils/utils.py) that support the above modules.
- Sample Data/Website Inputs (directories such as nykaafashion.com, www.tatacliq.com, etc.) – While not core to the architecture, they can be depicted as external sources or test inputs.

b. Relationships and Interactions:
- main.py serves as the central orchestrator and sequentially triggers:
- Reading configuration settings.
- Calling the scraper to load and scroll through pages.
- Passing the raw data to the processing module.
- Then handing off processed URLs to the saver module for output.
- From main.py to config, then to scraping.
- From scraping to processing.
- And finally from processing to saving.

c. Important Architectural Patterns:
- Modular Design: Each module (scraping, processing, saving) is separated, making it easier to maintain and extend.
- Parallel/Concurrent Processing: Emphasis on configurable parallel processing (MAX_WORKERS) to ensure the crawler can handle multiple domains simultaneously.
- Robustness and Scalability: The flow from discovery to storage, with logging at each stage, ensures that edge cases and exceptions are tracked.

d. Technologies and Frameworks:
- Python 3.x – Base language.
- Selenium – For browser automation in the scraping module.
- BeautifulSoup – For parsing HTML content.
- ChromeDriver – For controlling Chrome instances.
- Any additional libraries as listed in requirements.txt should be mentioned.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Flow
    A["Entry Point (main.py)"]:::orchestrator
    B["Configuration Module"]:::module
    C["Scraper Module"]:::module
    D["Processing Module"]:::module
    E["Saving Module"]:::module
    F["Utility Functions"]:::support

    A -->|"triggers"| B
    B -->|"supplies config to"| C
    C -->|"sends raw URLs to"| D
    D -->|"forwards processed data to"| E

    %% Utility Functions connections (dotted lines)
    B ---|"uses"| F
    C ---|"uses"| F
    D ---|"uses"| F
    E ---|"uses"| F

    %% Sample Website Inputs subgraph
    subgraph "Sample Website Inputs"
        S1["nykaafashion.com"]:::external
        S2["www.tatacliq.com"]:::external
        S3["www.virgio.com"]:::external
        S4["www.westside.com"]:::external
    end
    S1 --> C
    S2 --> C
    S3 --> C
    S4 --> C

    %% Logging & Data Storage subgraph
    subgraph "Logging & Data Storage"
        L1["product_links.json"]:::storage
        L2["endpoint.json"]:::storage
        L3["status_log.csv"]:::storage
        L4["initial_status_log.csv"]:::storage
    end
    E --> L1
    E --> L2
    E --> L3
    E --> L4

    %% Click Events
    click A "https://github.com/akhil298/web-crawler/blob/main/main.py"
    click B "https://github.com/akhil298/web-crawler/blob/main/config/config.py"
    click C "https://github.com/akhil298/web-crawler/blob/main/scraping/scraper.py"
    click D "https://github.com/akhil298/web-crawler/blob/main/processing/processor.py"
    click E "https://github.com/akhil298/web-crawler/blob/main/saving/saver.py"
    click F "https://github.com/akhil298/web-crawler/blob/main/utils/utils.py"
    click S1 "https://github.com/akhil298/web-crawler/blob/main/nykaafashion.com"
    click S2 "https://github.com/akhil298/web-crawler/blob/main/www.tatacliq.com"
    click S3 "https://github.com/akhil298/web-crawler/blob/main/www.virgio.com"
    click S4 "https://github.com/akhil298/web-crawler/blob/main/www.westside.com"
    click L1 "https://github.com/akhil298/web-crawler/blob/main/product_links.json"
    click L2 "https://github.com/akhil298/web-crawler/blob/main/endpoint.json"
    click L3 "https://github.com/akhil298/web-crawler/blob/main/status_log.csv"
    click L4 "https://github.com/akhil298/web-crawler/blob/main/initial_status_log.csv"

    %% Styles
    classDef orchestrator fill:#f4a261,stroke:#000,stroke-width:1px;
    classDef module fill:#2a9d8f,stroke:#000,stroke-width:1px;
    classDef support fill:#e9c46a,stroke:#000,stroke-width:1px;
    classDef external fill:#264653,stroke:#fff,stroke-width:1px;
    classDef storage fill:#e76f51,stroke:#000,stroke-width:1px;
```
