How lryanle/smare works
Project type and purpose
• This is a full-stack SaaS application named SMARE (Social Marketplace Automotive Risk Engine).
• Purpose: scrape social marketplaces (Craigslist, Facebook Marketplace), clean and score listings via multiple ML models to flag suspicious/insurance-fraud listings, persist results in MongoDB, and present a dashboard UI for end users.
File structure
• Top-level:
• frontend/ → Next.js + TypeScript client + API routes + Prisma ORM
• backend/ → Python pipeline: scrapers, cleaners, model scoring, orchestrator
• classification-chrome-extension/ → Chrome extension for labeling/classification
• documentation/ → project docs
• Key backend pieces:
• orchestrator.py → pipeline driver
• src/scrapers/ → Selenium & BeautifulSoup scrapers
• src/cleaners/ → data cleaning
• src/models/ → six scoring models & model manager
• src/utilities/database → MongoDB adapter (pymongo)
• src/sendGrid/notifs.py → email notifications
• Key frontend pieces:
• app/api/ → Next.js API routes as middleware
• lib/prisma.ts → Prisma client for MongoDB
• components/ → React UI (dashboard, datatable, shared UI)
• NextAuth for authentication
• Docker configs for local/dev/prod
README for additional insights
• Stack callouts: Next.js, Tailwind, Prisma, NextAuth, Selenium, BS4, Pandas, imbalanced-learn, Loguru, OpenAI API.
• High-level summary confirms pipeline segments: scraping → cleaning → modeling → dashboard.
Elements
• Component blocks with clear labels: “Browser UI”, “Next.js API Server”, “Prisma ORM”, “MongoDB Atlas”, “Python Scraper Service”, “Python Cleaner Service”, “ML Model Service”, “Orchestrator”, “OpenAI API”, “SendGrid”, “Chrome Extension”.
• Pipeline data flow (scrape → clean → model → store)
• Request/response flow (Browser → API → DB → Browser)
• External service calls
• Web layer (blue)
• API/database layer (green)
• Data-processing pipeline (orange)
• External services (purple)
• Chrome extension (gray)
Final advice
• Be very detailed: capture each pipeline step, each interaction.
• That each component block references the actual directory or file (e.g., backend/src/scrapers).