# charleskwakye/carindiascraping architecture

> Architecture diagram of the GitHub repository charleskwakye/carindiascraping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/charleskwakye/carindiascraping
- Repository: https://github.com/charleskwakye/carindiascraping

## Overview

##### Project type and purpose
- This is a standalone, command-line Python web-scraping tool (“open-source tool” style) whose sole purpose is to crawl autocarindia.com/cars, extract structured vehicle data, and persist it to CSV (with ancillary HTML dumps and screenshots for debugging).

##### File structure
- Top-level files:
- scraper.py – the main entry point and orchestrator.
- analyze_site.py – a helper module that inspects site structure (populates site_analysis.json).
- site_analysis.json – stores dropdown hierarchies or URL templates.
- requirements.txt – defines Python/Playwright dependencies.
- .gitignore – excludes venv, output directories, browser binaries.
- Implicit directories (created at runtime):
- output/ – timestamped CSV data.
- html_dumps/ – raw HTML for each page.
- screenshots/ – debug screenshots.

##### README for additional insights
- Uses Playwright to handle dynamic dropdowns and page navigation.
- Supports CLI flags (—limit-brands, —limit-models, —total-cars, etc.).
- Cleans old files before each run, incremental saving of partial results.

a. Main components
1. CLI Layer
2. Scraper Orchestrator (scraper.py)
3. Site Analyzer (analyze_site.py + site_analysis.json)
4. Browser Engine (Playwright)
5. Page Fetcher & Renderer
6. HTML Dump Module
7. Screenshot Module
8. Parser Module (extracts Brand, Model, Year, Fuel, Transmission, Engine, Mileage, Body Type, VIN, Price)
9. Data Exporter (CSV Writer + cleanup logic)
10. File System (output/, html_dumps/, screenshots/)
11. Logging/Error Handler

b. Relationships and data flow
- CLI Layer → invokes Scraper Orchestrator.
- Scraper Orchestrator reads CLI args and loads site_analysis.json (via Site Analyzer).
- Orchestrator calls Browser Engine to navigate to each brand/model page.
- Browser Engine → Page Fetcher: renders page and returns HTML.
- HTML Dump Module writes raw HTML to html_dumps/.
- Screenshot Module captures page snapshot → screenshots/.
- Parser Module processes HTML → Python dict per car.
- Data Exporter appends to CSV in output/ (cleanup module runs at job start).
- Logging/Error Handler monitors each step, writes to console/log file.

c. Architectural patterns & design principles
- Modular separation: Site analysis, scraping orchestration, parsing, and exporting are separate modules.
- Single-process, pipeline-style flow.
- Idempotent output: old files cleared, incremental writes to avoid data loss.
- Extensibility: Site Analyzer can be updated when pages change.

d. Key technologies & libraries
- Python 3.7+
- Playwright for browser automation
- argparse (CLI parsing)
- csv, os, json (stdlib modules)
- Fluent separation of I/O layers (file dumps vs. in-memory parsing)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Define nodes
    CLILayer["CLI Layer\n(supports flags)"]:::orchestration
    Orchestrator["Scraper Orchestrator"]:::orchestration
    SiteAnalyzer["Site Analyzer"]:::processing
    SiteConfig["Site Analysis Config\n(site_analysis.json)"]:::processing
    subgraph "Browser Engine\n(Playwright)"   
        direction TB
        BrowserEngine["Browser Engine"]:::io
        PageFetcher["Page Fetcher & Renderer"]:::io
    end
    HTMLDump["HTML Dump Module"]:::io
    ScreenshotMod["Screenshot Module"]:::io
    ParserMod["Parser Module"]:::processing
    Exporter["Data Exporter\n(CSV Writer & Cleanup)"]:::io
    subgraph "File System"
        direction TB
        OutputDir["output/"]:::filesystem
        HTMLDumpDir["html_dumps/"]:::filesystem
        ScreenshotDir["screenshots/"]:::filesystem
    end
    Logger["Logging/Error Handler"]:::processing
    Dependencies["Requirements"]:::docs
    GitIgnore[".gitignore"]:::docs
    Readme["README.md"]:::docs

    %% Connections
    CLILayer -->|reads flags| Orchestrator
    Orchestrator -->|loads JSON config| SiteAnalyzer
    SiteAnalyzer -->|reads/writes| SiteConfig
    Orchestrator -->|initializes| BrowserEngine
    BrowserEngine --> PageFetcher
    PageFetcher -->|HTML| HTMLDump
    HTMLDump -->|writes| HTMLDumpDir
    PageFetcher -->|screenshot| ScreenshotMod
    ScreenshotMod -->|writes| ScreenshotDir
    PageFetcher -->|HTML| ParserMod
    ParserMod -->|dict per car| Exporter
    Exporter -->|appends CSV rows| OutputDir
    Orchestrator -->|logs/errors| Logger

    %% Documentation nodes
    Dependencies
    GitIgnore
    Readme

    %% Click events
    click CLILayer "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click Orchestrator "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click SiteAnalyzer "https://github.com/charleskwakye/carindiascraping/blob/main/analyze_site.py"
    click SiteConfig "https://github.com/charleskwakye/carindiascraping/blob/main/site_analysis.json"
    click BrowserEngine "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click PageFetcher "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click HTMLDump "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click ScreenshotMod "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click ParserMod "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click Exporter "https://github.com/charleskwakye/carindiascraping/blob/main/scraper.py"
    click OutputDir "https://github.com/charleskwakye/carindiascraping/tree/main/output/"
    click HTMLDumpDir "https://github.com/charleskwakye/carindiascraping/tree/main/html_dumps/"
    click ScreenshotDir "https://github.com/charleskwakye/carindiascraping/tree/main/screenshots/"
    click Dependencies "https://github.com/charleskwakye/carindiascraping/blob/main/requirements.txt"
    click GitIgnore "https://github.com/charleskwakye/carindiascraping/blob/main/.gitignore"
    click Readme "https://github.com/charleskwakye/carindiascraping/blob/main/README.md"

    %% Styles
    classDef orchestration fill:#D6E4FF,stroke:#1D4ED8,color:#1D4ED8,stroke-width:1px;
    classDef io fill:#D1FAE5,stroke:#047857,color:#047857,stroke-width:1px;
    classDef processing fill:#FEF3C7,stroke:#B45309,color:#B45309,stroke-width:1px;
    classDef filesystem fill:#E5E7EB,stroke:#6B7280,color:#6B7280,stroke-width:1px;
    classDef docs fill:#EEF2FF,stroke:#4338CA,color:#4338CA,stroke-width:1px;
```
