How yesidmg/smart-search works
Project Type and Purpose
• This is a full-stack, microservices-based web application (“Smart Search”) whose goal is to collect user preferences, process them through an LLM, scrape product URLs, and return personalized recommendations.
Key Architectural Components
a. Frontend Service (React + Vite)
• Lives in the front/ directory.
• Captures user input via UI and publishes messages to RabbitMQ (solicitudes).
• Subscribes to RabbitMQ queues (respuestas for status updates and scraped_urls_queue for final recommendations).
b. Backend Workers (back/)
• Request Processor (index.py):
– Consumer of solicitudes queue.
– Persists user/test data to PostgreSQL.
– Publishes a status message to respuestas.
– Publishes a user-profile message to peticiones_ia.
• Scraper (scraper.py):
– Consumer of scrapper_peticiones_queue.
– Performs web scraping on MercadoLibre Colombia.
– Publishes scraped URLs to scraped_urls_queue.
• These two processes run under Supervisor.
c. AI Service (ai_service/)
• Flask Proxy (app.py): health endpoint + direct proxy to OpenRouter API.
• RabbitMQ Consumer (rabbitmq_client.py):
– Listens on peticiones_ia.
– Calls the LLM via openrouter_client.py.
– Publishes generated search terms to scrapper_peticiones_queue.
d. Message Broker (RabbitMQ)
• Manages asynchronous message queues: solicitudes, respuestas, peticiones_ia, scrapper_peticiones_queue, scraped_urls_queue.
e. Database (PostgreSQL)
• Stores user data, tests, requests. Initialized by Postgres/init.sql.
f. Reverse Proxy (Traefik)
• Routes HTTP traffic to frontend and can expose AI service.
Interactions & Data Flows
1. User → Frontend UI (HTTP)
2. Frontend → RabbitMQ (solicitudes) (AMQP)
3. Request Processor → PostgreSQL (SQL)
4. Request Processor → RabbitMQ (respuestas & peticiones_ia) (AMQP)
5. AI Consumer → OpenRouter LLM (HTTP)
6. AI Consumer → RabbitMQ (scrapper_peticiones_queue) (AMQP)
7. Scraper → External Website (HTTP)
8. Scraper → RabbitMQ (scraped_urls_queue) (AMQP)
9. Frontend → RabbitMQ (listening to respuestas & scraped_urls_queue)
Architectural Patterns & Design Principles
• Microservices with clear service boundaries.
• Asynchronous, event-driven communication via message broker.
• Separation of concerns: UI, business logic (workers), ML orchestration, data persistence.
• Containerization (Docker Compose) for orchestration and process isolation.
• Process supervision (Supervisor) for worker resilience.