How laporeon/meliscraper works
Project type and purpose
• This is a single-service backend API (a Node.js/Express microservice) whose sole responsibility is to scrape Mercado Libre best-seller listings, persist them in PostgreSQL via Prisma, and expose REST endpoints (plus Swagger docs) for clients to query scrapes, categories, and products.
File structure
• Top-level files:
• package.json, .env.example – dependencies and environment settings
• prisma/ – schema.prisma and migration history (defines Category, Product, Scraping tables and relationships)
• src/ – application code:
– server.js (Express app entrypoint)
– routes/ (express.Router definitions)
– controllers/ (handler functions orchestrating requests)
– utils/ (scraper logic, slugify, date formatting, logging, constants)
– database/prisma.js (Prisma client setup)
• docs/ – Swagger configuration: paths and schemas
README for additional insights
• Technical stack: Node 20, PostgreSQL 14, Prisma 6
• Usage patterns:
• POST /scrapings → trigger scrape of top-10 products per category (only one scrape per date allowed)
• GET endpoints for categories, products by category, and scrape by date
• Swagger UI at /documentation
a. Main components:
• Clients (e.g. browser or REST client & Swagger UI)
• Express API Server
– Routing Layer (routes/*.js)
– Controller Layer (controllers/*.js)
– Business Logic / Scraper Utilities (utils/scraper.js, utils/date.js, etc.)
– Data Access Layer (database/prisma.js using Prisma Client)
• PostgreSQL Database
• External Source: Mercado Libre Website (HTML pages to scrape)
b. Relationships & interactions:
1. Client → HTTP Request → Express Routing
2. Routing → Controller → invokes either:
– Prisma Client queries/updates → Database
– Scraper Utility → HTTP requests to Mercado Libre → parse data → Prisma Client → Database
3. Controller → HTTP Response → Client
4. Swagger UI served by Express → Client
c. Architectural patterns & design principles:
• Layered architecture (Presentation → Controllers → Services/Utils → Data Access → Database)
• Single-responsibility modules (each controller handles one resource, utils handle cross-cutting concerns)
• Prisma as ORM/ODM for type-safe DB access
d. Key technologies & libraries:
• Node.js & Express.js for the web server
• Prisma Client for DB interaction
• PostgreSQL as the relational datastore
• Axios or built-in HTTP for scraping (abstracted in utils/scraper.js)
• Swagger (docs/swagger.js + docs/paths & docs/schemas) for API documentation
Diagramming recommendations
• “Client & Swagger UI”
• “Express Router”
• “Controllers”
• “Scraping Service / Utils”
• “Prisma Client (Data Access)”
• “PostgreSQL”
• “External: Mercado Libre”
• Client → Router → Controllers → [Data Access → DB]
• Controllers → Scraper Utils → External Site → Scraper Utils → Data Access → DB
• Server → Swagger UI → Client
• API components (blue)
• Business logic / scraper (green)
• Data access / database (orange)
• External system (gray)