# mozartmarin/webscraping-2025-jornada-aovivo architecture

> Architecture diagram of the GitHub repository mozartmarin/webscraping-2025-jornada-aovivo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mozartmarin/webscraping-2025-jornada-aovivo
- Repository: https://github.com/mozartmarin/webscraping-2025-jornada-aovivo

## Overview

##### Project Type and Purpose
- This is a data‐engineering project implementing an end-to-end ETL pipeline for scraping Samsung product data from Mercado Livre, transforming it, loading it into a local database, and exposing it via a dashboard.

##### File Structure Breakdown
- Top-level files and folders:
– data/ : holds raw JSONL and the SQLite database
– pic/ : assets (logo for the dashboard)
– requirements.txt : Python dependencies
– src/ : all source code, split into three subprojects
- src/extraction/coleta : a Scrapy project (“coleta”) with spiders that crawl Mercado Livre, outputting to data/data.jsonl or directly into mercadolivre.db
- src/transformLoad/main.py : a standalone ETL script that reads raw data (JSONL or DB), applies cleaning/normalization, and writes back to the database
- src/dashboard/app.py : a dashboard application (likely Flask or Dash) that reads the transformed data from mercadolivre.db and serves interactive visualizations

##### Key Components
a. External Service
– Mercado Livre website (HTTP endpoints)
b. Extraction Layer
– Scrapy engine & spiders (src/extraction/coleta)
– Outputs: data/data.jsonl and/or mercadolivre.db
c. Storage (Raw)
– Local JSONL file (data/data.jsonl)
– SQLite database (data/mercadolivre.db)
d. Transform & Load Layer
– ETL script (src/transformLoad/main.py)
– Reads raw data, applies transformations, writes back to mercadolivre.db
e. Dashboard Layer
– Web server (src/dashboard/app.py)
– Reads from the transformed SQLite DB and renders charts/UI

##### Relationships and Data Flow
2. “Scrapy Spiders” → “Raw Storage” (JSONL & SQLite)
3. “ETL Script” reads from “Raw Storage” → writes to “Transformed Storage” (re-uses SQLite)
4. “Dashboard App” reads from “Transformed Storage” → “End User” in the browser

##### Architectural Patterns & Design Principles
– Layered ETL pipeline (Extraction → Staging → Transformation → Presentation)
– Single responsibility per component: spiders only scrape, main.py only transforms, app.py only serves
– Configuration/externalization in scrapy.cfg and requirements.txt
– Modular code organization under src/ for clear separation

##### Technologies & Frameworks
– Scrapy (web crawling framework)
– Python scripts for transformation (pandas, sqlite3, etc.)
– SQLite (lightweight on-disk database)
– Flask or Dash (for dashboard; verify in app.py)
– JSON Lines (data.jsonl) for raw dumps

##### Detail Level
- Important config files (scrapy.cfg, requirements.txt) at each layer
- That the dashboard reads directly from the transformed SQLite DB

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Service
    mercado["Mercado Livre\n(HTTP)"]:::external

    %% Extraction Layer
    subgraph "Extraction Layer"
        scrapyProj["Scrapy Project\n(src/extraction/coleta)"]:::extraction
        spiders["Spiders\n(src/extraction/coleta/spiders)"]:::extraction
        cfg["scrapy.cfg"]:::config
    end

    %% Raw Storage
    subgraph "Raw Storage"
        json["data/data.jsonl"]:::storage
        rawdb["data/mercadolivre.db\n(Raw)"]:::storage
    end

    %% Transform & Load Layer
    subgraph "Transform & Load Layer"
        etl["ETL Script\n(src/transformLoad/main.py)"]:::etl
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        dashboard["Dashboard App\n(src/dashboard/app.py)"]:::presentation
        assets["Assets\n(pic/)"]:::presentation
    end

    %% Configurations & Docs
    subgraph "Configurations & Docs"
        req["requirements.txt"]:::config
        readme["README.md"]:::config
        gitignore[".gitignore"]:::config
    end

    %% End User
    endUser["End User\n(Browser)"]:::external

    %% Data Flows
    mercado -->|"HTTP"| scrapyProj
    scrapyProj --> spiders
    spiders -->|"JSONL"| json
    spiders -->|"SQL inserts"| rawdb
    rawdb -->|"raw data"| etl
    json -->|"raw data"| etl
    etl -->|"writes transformed data"| rawdb
    rawdb -->|"reads transformed data"| dashboard
    dashboard -->|"serves UI"| endUser

    %% Click Events
    click scrapyProj "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/tree/main/src/extraction/coleta"
    click spiders "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/tree/main/src/extraction/coleta/spiders"
    click cfg "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/src/extraction/scrapy.cfg"
    click json "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/data/data.jsonl"
    click rawdb "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/data/mercadolivre.db"
    click etl "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/src/transformLoad/main.py"
    click dashboard "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/src/dashboard/app.py"
    click assets "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/tree/main/pic/"
    click req "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/requirements.txt"
    click readme "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/README.md"
    click gitignore "https://github.com/mozartmarin/webscraping-2025-jornada-aovivo/blob/main/.gitignore"

    %% Styles
    classDef extraction fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef etl fill:#ffe0b2,stroke:#333,stroke-width:1px;
    classDef presentation fill:#c8e6c9,stroke:#333,stroke-width:1px;
    classDef storage fill:#e0e0e0,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef external fill:#e1bee7,stroke:#333,stroke-width:1px;
    classDef config fill:#f5f5f5,stroke:#333,stroke-width:1px;
```
