How mozartmarin/webscraping-2025-jornada-aovivo works
Project Type and Purpose
• This is a data‐engineering project implementing an end-to-end ETL pipeline for scraping Samsung product data from Mercado Livre, transforming it, loading it into a local database, and exposing it via a dashboard.
File Structure Breakdown
• Top-level files and folders:
– data/ : holds raw JSONL and the SQLite database
– pic/ : assets (logo for the dashboard)
– requirements.txt : Python dependencies
– src/ : all source code, split into three subprojects
• src/extraction/coleta : a Scrapy project (“coleta”) with spiders that crawl Mercado Livre, outputting to data/data.jsonl or directly into mercadolivre.db
• src/transformLoad/main.py : a standalone ETL script that reads raw data (JSONL or DB), applies cleaning/normalization, and writes back to the database
• src/dashboard/app.py : a dashboard application (likely Flask or Dash) that reads the transformed data from mercadolivre.db and serves interactive visualizations
Key Components
a. External Service
– Mercado Livre website (HTTP endpoints)
b. Extraction Layer
– Scrapy engine & spiders (src/extraction/coleta)
– Outputs: data/data.jsonl and/or mercadolivre.db
c. Storage (Raw)
– Local JSONL file (data/data.jsonl)
– SQLite database (data/mercadolivre.db)
d. Transform & Load Layer
– ETL script (src/transformLoad/main.py)
– Reads raw data, applies transformations, writes back to mercadolivre.db
e. Dashboard Layer
– Web server (src/dashboard/app.py)
– Reads from the transformed SQLite DB and renders charts/UI
Relationships and Data Flow
2. “Scrapy Spiders” → “Raw Storage” (JSONL & SQLite)
3. “ETL Script” reads from “Raw Storage” → writes to “Transformed Storage” (re-uses SQLite)
4. “Dashboard App” reads from “Transformed Storage” → “End User” in the browser
Architectural Patterns & Design Principles
– Layered ETL pipeline (Extraction → Staging → Transformation → Presentation)
– Single responsibility per component: spiders only scrape, main.py only transforms, app.py only serves
– Configuration/externalization in scrapy.cfg and requirements.txt
– Modular code organization under src/ for clear separation
Technologies & Frameworks
– Scrapy (web crawling framework)
– Python scripts for transformation (pandas, sqlite3, etc.)
– SQLite (lightweight on-disk database)
– Flask or Dash (for dashboard; verify in app.py)
– JSON Lines (data.jsonl) for raw dumps
Detail Level
• Important config files (scrapy.cfg, requirements.txt) at each layer
• That the dashboard reads directly from the transformed SQLite DB