# bharat-agrawal-05/data-scrape architecture

> Architecture diagram of the GitHub repository bharat-agrawal-05/data-scrape, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bharat-agrawal-05/data-scrape
- Repository: https://github.com/bharat-agrawal-05/data-scrape

## Overview

##### Project Type and Purpose
- This is a scripted data‐harvesting pipeline (an open‐source scraping & structuring tool), not a web service or full‐stack app.
- Purpose: take a list of URLs (links.txt), crawl & download documents (HTML, PDF), extract text & metadata, and write structured JSON.

##### File Structure
- Top‐level files:
- links.txt – input list of target URLs
- main.py – orchestrates the pipeline
- main.ipynb – exploratory notebook/prototype
- requirements.txt – Python dependencies
- README.md – project overview & instructions
- Runtime folders (created at run‐time):
- downloads/ – raw HTML & PDF files
- json_output/ – per‐document JSON files

##### Extract Key Workflow from README
- Input: links.txt
- Pipeline steps:
a. Crawl each URL
b. Download supported files (HTML & PDF)
c. Extract metadata (title, authors, year, etc.)
d. For PDFs: convert pages to images (poppler + pdf2image) → OCR with Tesseract
e. Save raw files in downloads/
f. Emit one JSON document per input in json_output/

##### Main Components & Their Interactions
a. Input Module
- Reads links.txt
- Outputs list of URLs
b. Crawl & Download Module
- Uses HTTP client (e.g., requests)
- Fetches HTML pages and PDF binaries
- Writes to downloads/
c. File Processing Module
- HTML parser (e.g., BeautifulSoup) → extracts in‐page text & metadata
- PDF‐to‐image converter (poppler/pdf2image) → image files
- OCR engine (Tesseract via pytesseract) → extracted text
d. Metadata Extraction Component
- Pulls fields (title, authors, year, language, checksum, timestamp)
e. Output Module
- Structures full results into JSON
- Writes JSON to json_output/
f. External Dependencies
- Poppler (system binary)
- Tesseract OCR (system binary)

##### Architectural Patterns & Design Principles
- Linear ETL‐style pipeline
- Single orchestrator (main.py) coordinating each stage
- Clear separation:
- Input (links list)
- Processing steps (crawl, download, extract)
- Storage (raw & structured)
- Extensibility points: new file types, alternate OCR engines

##### Technologies & Libraries to Annotate
- Python 3.x
- requests or urllib (HTTP)
- BeautifulSoup (HTML parsing)
- pdf2image + poppler (PDF → image)
- pytesseract + Tesseract OCR (image → text)
- hashlib (checksum)
- JSON library (serialization)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Source
    subgraph "Data Source"
        Links["links.txt"]:::input
    end

    %% Orchestrator
    Main["Orchestrator\n(main.py)"]:::processing

    %% Pipeline Modules
    Crawler["Crawler & Downloader"]:::processing
    subgraph "File Processing"
        HTMLParser["HTML Parser\n(BeautifulSoup)"]:::processing
        PDFConverter["PDF→Image Converter\n(pdf2image)"]:::processing
        OCREngine["OCR Engine\n(Tesseract)"]:::processing
    end
    Metadata["Metadata Extractor"]:::processing
    Output["JSON Writer"]:::storage

    %% Storage Outputs
    Downloads["downloads/"]:::storage
    JSONDir["json_output/"]:::storage

    %% External Dependencies
    Poppler["Poppler"]:::external
    TesseractBinary["Tesseract OCR"]:::external

    %% Documentation & Config
    Notebook["main.ipynb"]:::docs
    Requirements["requirements.txt"]:::docs
    README["README.md"]:::docs

    %% Data Flow
    Links -->|"URL list"| Main
    Main -->|"initiate crawl"| Crawler
    Crawler -->|"HTML, PDF"| Downloads
    Crawler -->|"HTML pages"| HTMLParser
    Crawler -->|"PDF binaries"| PDFConverter
    PDFConverter -->|"images"| OCREngine
    HTMLParser -->|"raw text"| Metadata
    OCREngine -->|"raw text"| Metadata
    Metadata -->|"metadata + text"| Output
    Output -->|"JSON files"| JSONDir

    %% External Invocations
    Poppler -->|"pdf2image"| PDFConverter
    TesseractBinary -->|"OCR"| OCREngine

    %% Docs & Config Side Box
    Notebook -.-> Main
    Requirements -.-> Main
    README -.-> Main

    %% Click Events
    click Links "https://github.com/bharat-agrawal-05/data-scrape/blob/main/links.txt"
    click Main "https://github.com/bharat-agrawal-05/data-scrape/blob/main/main.py"
    click Notebook "https://github.com/bharat-agrawal-05/data-scrape/blob/main/main.ipynb"
    click Requirements "https://github.com/bharat-agrawal-05/data-scrape/blob/main/requirements.txt"
    click README "https://github.com/bharat-agrawal-05/data-scrape/blob/main/README.md"
    click Downloads "https://github.com/bharat-agrawal-05/data-scrape/tree/main/downloads/"
    click JSONDir "https://github.com/bharat-agrawal-05/data-scrape/tree/main/json_output/"

    %% Styles
    classDef input fill:#add8e6,stroke:#000,stroke-width:1px
    classDef processing fill:#90ee90,stroke:#000,stroke-width:1px
    classDef storage fill:#ffa500,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef docs fill:#dda0dd,stroke:#000,stroke-width:1px
```
