How charleskwakye/carindiascraping works
Project type and purpose
• This is a standalone, command-line Python web-scraping tool (“open-source tool” style) whose sole purpose is to crawl autocarindia.com/cars, extract structured vehicle data, and persist it to CSV (with ancillary HTML dumps and screenshots for debugging).
File structure
• Top-level files:
• scraper.py – the main entry point and orchestrator.
• analyze_site.py – a helper module that inspects site structure (populates site_analysis.json).
• site_analysis.json – stores dropdown hierarchies or URL templates.
• requirements.txt – defines Python/Playwright dependencies.
• .gitignore – excludes venv, output directories, browser binaries.
• Implicit directories (created at runtime):
• output/ – timestamped CSV data.
• html_dumps/ – raw HTML for each page.
• screenshots/ – debug screenshots.
README for additional insights
• Uses Playwright to handle dynamic dropdowns and page navigation.
• Supports CLI flags (—limit-brands, —limit-models, —total-cars, etc.).
• Cleans old files before each run, incremental saving of partial results.
a. Main components
1. CLI Layer
2. Scraper Orchestrator (scraper.py)
3. Site Analyzer (analyze_site.py + site_analysis.json)
4. Browser Engine (Playwright)
5. Page Fetcher & Renderer
6. HTML Dump Module
7. Screenshot Module
8. Parser Module (extracts Brand, Model, Year, Fuel, Transmission, Engine, Mileage, Body Type, VIN, Price)
9. Data Exporter (CSV Writer + cleanup logic)
10. File System (output/, html_dumps/, screenshots/)
11. Logging/Error Handler
b. Relationships and data flow
• CLI Layer → invokes Scraper Orchestrator.
• Scraper Orchestrator reads CLI args and loads site_analysis.json (via Site Analyzer).
• Orchestrator calls Browser Engine to navigate to each brand/model page.
• Browser Engine → Page Fetcher: renders page and returns HTML.
• HTML Dump Module writes raw HTML to html_dumps/.
• Screenshot Module captures page snapshot → screenshots/.
• Parser Module processes HTML → Python dict per car.
• Data Exporter appends to CSV in output/ (cleanup module runs at job start).
• Logging/Error Handler monitors each step, writes to console/log file.
c. Architectural patterns & design principles
• Modular separation: Site analysis, scraping orchestration, parsing, and exporting are separate modules.
• Single-process, pipeline-style flow.
• Idempotent output: old files cleared, incremental writes to avoid data loss.
• Extensibility: Site Analyzer can be updated when pages change.
d. Key technologies & libraries
• Python 3.7+
• Playwright for browser automation
• argparse (CLI parsing)
• csv, os, json (stdlib modules)
• Fluent separation of I/O layers (file dumps vs. in-memory parsing)