How shxntanu/email-classifier works
Project Type & Purpose
• This is a full-stack solution designed for auto-classifying and routing emails. It uses a combination of email inbox monitoring, distributed task queues (using Celery and RabbitMQ), encryption/decryption, and large language model (LLM) analysis (via Ollama with models like Qwen 4b and Mixtral 8x7b) to determine optimal email routing.
• The system is built as a multi-component service that involves email ingestion, ML-driven classification/summarization, attachments handling, and routing via SMTP.
Analyzing the File Structure
• Multiple directories are present indicating separation of responsibilities:
• The “hackathon/ML” folder contains Python scripts for ML processing (e.g., app.py, tree.py, llm.py) and data files (CSV and JSON) used for training or routing.
• The “hackathon/client” folder is a Next.js-based frontend, suggesting a web interface for monitoring or manual interactions.
• The “hackathon/server” directory houses the backend logic (e.g., main.py, lib with modules such as forward.py and summarize.py, and data JSON files defining the routing hierarchy).
• There are “testing” directories for spam detection, dataset testing, and script-testing which can be considered supplementary validation components.
• The “src” top-level directory replicates core functionality (app.py, data, library modules, and tests).
• The presence of configuration files (like .env.example, tailwind.config.js, package.json) and build-related assets means the system is designed for containerized and production deployment.
To accurately represent the architecture, include the following elements and guidelines:
a. Main Components
• Email Receiver Service: A continuously running Python service that uses IMAP protocol to monitor a central email inbox.
• Parsing & Encryption Module: Handles email parsing, attachment extraction, and encryption before forwarding.
• Server/Decryption & LLM Processing: This service decrypts incoming emails, runs them through the LLM (Mixtral 8x7b or Qwen 4b via Ollama) for context extraction and summarization.
• Summarization Module: A lighter model (BART or Qwen 4b) that outputs a summary.
• Routing Engine: Uses JSON-based hierarchy (from data files like rag.json and heirarchy.json) to determine the correct destination team or department.
• Mail Forwarding Service: Uses SMTP for sending emails, with added features like watermarking for product confirmation.
• Frontend Dashboard (Client): A Next.js-based UI for feedback, monitoring, and possibly retraining the model.
b. Relationships and Interactions
• From Email Receiver → to the Task Queue.
• From Task Queue → to the Parsing & Encryption Module.
• From Parsing → to the Server/Decryption & LLM Processing.
• From LLM Processing → to the Summarization Module.
• From both LLM and Summarization outputs → to the Routing Engine.
• From Routing Engine → to Mail Forwarding.
• Interactions with external systems: IMAP and SMTP servers, the RabbitMQ broker, and the Ollama model service.
c. Architectural Patterns & Design Principles
• Decoupling of components using message brokers (RabbitMQ) and distributed task processing (Celery), which provides horizontal scalability.
• Modular design: separate directories indicate that the ML logic, API services, and web front-end are loosely coupled.
d. Relevant Technologies, Frameworks & Libraries
• Python (core language)
• Anaconda/Poetry for dependency management.
• Celery (distributed task queue) and RabbitMQ (message broker)
• Ollama for on-device LLM inference (Qwen 4b and Mixtral 8x7b)
• Next.js (frontend UI)
• Additional libraries for encryption, summarization, and attachment handling
Instructing the Principal Software Engineer
• Finally, consider adding a “Feedback Loop” section to denote the retraining and fine-tuning process based on user or system feedback.