How bharat-agrawal-05/data-scrape works
Project Type and Purpose
• This is a scripted data‐harvesting pipeline (an open‐source scraping & structuring tool), not a web service or full‐stack app.
• Purpose: take a list of URLs (links.txt), crawl & download documents (HTML, PDF), extract text & metadata, and write structured JSON.
File Structure
• Top‐level files:
• links.txt – input list of target URLs
• main.py – orchestrates the pipeline
• main.ipynb – exploratory notebook/prototype
• requirements.txt – Python dependencies
• README.md – project overview & instructions
• Runtime folders (created at run‐time):
• downloads/ – raw HTML & PDF files
• json_output/ – per‐document JSON files
Extract Key Workflow from README
• Input: links.txt
• Pipeline steps:
a. Crawl each URL
b. Download supported files (HTML & PDF)
c. Extract metadata (title, authors, year, etc.)
d. For PDFs: convert pages to images (poppler + pdf2image) → OCR with Tesseract
e. Save raw files in downloads/
f. Emit one JSON document per input in json_output/
Main Components & Their Interactions
a. Input Module
• Reads links.txt
• Outputs list of URLs
b. Crawl & Download Module
• Uses HTTP client (e.g., requests)
• Fetches HTML pages and PDF binaries
• Writes to downloads/
c. File Processing Module
• HTML parser (e.g., BeautifulSoup) → extracts in‐page text & metadata
• PDF‐to‐image converter (poppler/pdf2image) → image files
• OCR engine (Tesseract via pytesseract) → extracted text
d. Metadata Extraction Component
• Pulls fields (title, authors, year, language, checksum, timestamp)
e. Output Module
• Structures full results into JSON
• Writes JSON to json_output/
f. External Dependencies
• Poppler (system binary)
• Tesseract OCR (system binary)
Architectural Patterns & Design Principles
• Linear ETL‐style pipeline
• Single orchestrator (main.py) coordinating each stage
• Clear separation:
• Input (links list)
• Processing steps (crawl, download, extract)
• Storage (raw & structured)
• Extensibility points: new file types, alternate OCR engines
Technologies & Libraries to Annotate
• Python 3.x
• requests or urllib (HTTP)
• BeautifulSoup (HTML parsing)
• pdf2image + poppler (PDF → image)
• pytesseract + Tesseract OCR (image → text)
• hashlib (checksum)
• JSON library (serialization)