# laporeon/meliscraper architecture

> Architecture diagram of the GitHub repository laporeon/meliscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/laporeon/meliscraper
- Repository: https://github.com/laporeon/meliscraper

## Overview

##### Project type and purpose
- This is a single-service backend API (a Node.js/Express microservice) whose sole responsibility is to scrape Mercado Libre best-seller listings, persist them in PostgreSQL via Prisma, and expose REST endpoints (plus Swagger docs) for clients to query scrapes, categories, and products.

##### File structure
- Top-level files:
- package.json, .env.example – dependencies and environment settings
- prisma/ – schema.prisma and migration history (defines Category, Product, Scraping tables and relationships)
- src/ – application code:
– server.js (Express app entrypoint)
– routes/ (express.Router definitions)
– controllers/ (handler functions orchestrating requests)
– utils/ (scraper logic, slugify, date formatting, logging, constants)
– database/prisma.js (Prisma client setup)
- docs/ – Swagger configuration: paths and schemas

##### README for additional insights
- Technical stack: Node 20, PostgreSQL 14, Prisma 6
- Usage patterns:
- POST /scrapings → trigger scrape of top-10 products per category (only one scrape per date allowed)
- GET endpoints for categories, products by category, and scrape by date
- Swagger UI at /documentation

a. Main components:
- Clients (e.g. browser or REST client & Swagger UI)
- Express API Server
– Routing Layer (routes/*.js)
– Controller Layer (controllers/*.js)
– Business Logic / Scraper Utilities (utils/scraper.js, utils/date.js, etc.)
– Data Access Layer (database/prisma.js using Prisma Client)
- PostgreSQL Database
- External Source: Mercado Libre Website (HTML pages to scrape)
b. Relationships & interactions:
1. Client → HTTP Request → Express Routing
2. Routing → Controller → invokes either:
– Prisma Client queries/updates → Database
– Scraper Utility → HTTP requests to Mercado Libre → parse data → Prisma Client → Database
3. Controller → HTTP Response → Client
4. Swagger UI served by Express → Client
c. Architectural patterns & design principles:
- Layered architecture (Presentation → Controllers → Services/Utils → Data Access → Database)
- Single-responsibility modules (each controller handles one resource, utils handle cross-cutting concerns)
- Prisma as ORM/ODM for type-safe DB access
d. Key technologies & libraries:
- Node.js & Express.js for the web server
- Prisma Client for DB interaction
- PostgreSQL as the relational datastore
- Axios or built-in HTTP for scraping (abstracted in utils/scraper.js)
- Swagger (docs/swagger.js + docs/paths & docs/schemas) for API documentation

##### Diagramming recommendations
- “Client & Swagger UI”
- “Express Router”
- “Controllers”
- “Scraping Service / Utils”
- “Prisma Client (Data Access)”
- “PostgreSQL”
- “External: Mercado Libre”
- Client → Router → Controllers → [Data Access → DB]
- Controllers → Scraper Utils → External Site → Scraper Utils → Data Access → DB
- Server → Swagger UI → Client
- API components (blue)
- Business logic / scraper (green)
- Data access / database (orange)
- External system (gray)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client & Swagger UI"
        Client["Client (Browser/REST)"]:::client
        SwaggerUI["Swagger UI"]:::client
    end

    subgraph "Express API Server"
        Router["Express Routing Layer"]:::api

        subgraph "Controllers"
            CatCtrl["Category Controller"]:::controller
            ProdCtrl["Product Controller"]:::controller
            ScrapCtrl["Scraper Controller"]:::controller
        end

        subgraph "Business Logic / Utils"
            ScraperUtil["Scraper Utility"]:::logic
            DateUtil["Date Formatting"]:::logic
            SlugUtil["Slug Generation"]:::logic
            Logger["Logging"]:::logic
            Constants["Constants"]:::logic
        end

        DAL["Prisma Client (Data Access)"]:::data
    end

    DB["PostgreSQL Database"]:::database
    External["Mercado Libre (External Site)"]:::external

    %% Connections
    Client -->|"HTTP Request"| Router
    SwaggerUI -->|"HTTP Request"| Router
    Router -->|"route: /categories"| CatCtrl
    Router -->|"route: /products"| ProdCtrl
    Router -->|"route: /scrapings"| ScrapCtrl
    Router -->|"route: /documentation"| SwaggerUI

    CatCtrl -->|"DB Query"| DAL
    ProdCtrl -->|"DB Query"| DAL
    ScrapCtrl -->|"invoke Scraper"| ScraperUtil
    ScrapCtrl -->|"use Date/Slug/Logger"| DateUtil
    ScrapCtrl -->|"use Date/Slug/Logger"| SlugUtil
    ScrapCtrl -->|"use Date/Slug/Logger"| Logger
    ScrapCtrl -->|"use Constants"| Constants

    ScraperUtil -->|"HTTP Request"| External
    External -->|"HTML Response"| ScraperUtil
    ScraperUtil -->|"DB Upsert"| DAL

    DAL -->|"SQL Queries"| DB

    CatCtrl -->|"HTTP Response"| Client
    ProdCtrl -->|"HTTP Response"| Client
    ScrapCtrl -->|"HTTP Response"| Client

    %% Click Events
    click Client "https://github.com/laporeon/meliscraper/tree/main//"
    click SwaggerUI "https://github.com/laporeon/meliscraper/tree/main//"
    click Router "https://github.com/laporeon/meliscraper/blob/main/src/routes/index.js"
    click CatCtrl "https://github.com/laporeon/meliscraper/blob/main/src/controllers/category-controller.js"
    click ProdCtrl "https://github.com/laporeon/meliscraper/blob/main/src/controllers/product-controller.js"
    click ScrapCtrl "https://github.com/laporeon/meliscraper/blob/main/src/controllers/scraper-controller.js"
    click ScraperUtil "https://github.com/laporeon/meliscraper/blob/main/src/utils/scraper.js"
    click DateUtil "https://github.com/laporeon/meliscraper/blob/main/src/utils/date.js"
    click SlugUtil "https://github.com/laporeon/meliscraper/blob/main/src/utils/slugify.js"
    click Logger "https://github.com/laporeon/meliscraper/blob/main/src/utils/logger.js"
    click Constants "https://github.com/laporeon/meliscraper/blob/main/src/utils/constants.js"
    click DAL "https://github.com/laporeon/meliscraper/blob/main/src/database/prisma.js"
    click DB "https://github.com/laporeon/meliscraper/blob/main//prisma/schema.prisma"
    click External "https://github.com/laporeon/meliscraper/tree/main//"

    %% Styles
    classDef client fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef api fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef controller fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef logic fill:#d4edda,stroke:#333,stroke-width:1px
    classDef data fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef database fill:#f8d7da,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
```
