# akshatv07/email-automation architecture

> Architecture diagram of the GitHub repository akshatv07/email-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akshatv07/email-automation
- Repository: https://github.com/akshatv07/email-automation

## Overview

##### Project type and purpose
- This is a backend-only application (a CLI/service) for automating ticket resolution via semantic search. It ingests CSV and Excel knowledge-base data, indexes it in a Milvus vector store, and then uses a semantic response engine to generate email replies based on incoming tickets.

##### File structure
- Top-level scripts and entry points
- main.py – orchestrates end-to-end flow
- bedrock.py – integration with AWS Bedrock (optional LLM provider)
- email_responder.py, retrieve_email.py, manual_email_processor.py – handle email I/O and manual overrides
- utility scripts (list_collections.py, inspect_collections.py, delete_all_collections.py) – Milvus collection management
- config/
- settings.py – global configuration (Milvus host/port, email server, file paths)
- data/
- datadb.csv – ticket metadata
- kb_sheets/*.xlsx – raw knowledge-base content
- core/
- data_db_processor.py – transforms CSV metadata into structured form
- vector_ingestion_engine.py – converts KB into vector embeddings and pushes to Milvus
- (semantic_response_engine.py per README) – vector query + response extraction
- templates/templates/
- dozens of HTML templates keyed by ticket category for email responses
- tests and examples
- test_ingest.py, test_robust_query.py – unit tests
- example_integration.py – sample usage

##### Key insights from README
- Milvus runs in Docker via docker-compose.yml
- Python 3.8+, venv, pip-installable requirements.txt
- SemanticResponseEngine is the core API for querying
- End-to-end: ingest → semantic store → query → respond

##### Main components
a. Data Sources
- CSV metadata (data/datadb.csv)
- Excel KB sheets (data/kb_sheets/*.xlsx)
b. Data Processing & Ingestion
- DataDB Processor (core/data_db_processor.py)
- Vector Ingestion Engine (core/vector_ingestion_engine.py)
c. Storage & Index
- Milvus Vector Store (Docker container)
- Metadata store (could be local files or relational DB)
d. Semantic Response Engine
- Semantic query module (core/semantic_response_engine.py)
- Embedding model (OpenAI/AWS Bedrock via bedrock.py)
e. Email Handling
- Retrieve incoming emails (retrieve_email.py)
- Route tickets to semantic engine
- Format responses via Jinja-style HTML templates
- Send replies (email_responder.py)
- Manual override tool (manual_email_processor.py)
f. Utilities & Admin
- Collection management scripts
- Configuration module (config/settings.py)

##### Relationships and interactions
- Ingestion pipeline: raw files → DataDB Processor → Vector Ingestion → Milvus
- Query flow: incoming email → SemanticResponseEngine → template lookup → Email Responder → SMTP
- Admin tools talk directly to Milvus for maintenance

##### Architectural patterns and design principles
- Separation of concerns: ingestion vs query vs email I/O
- Pipeline architecture for data flow
- Single responsibility modules for each stage
- Extensibility: new embedding models (bedrock.py), new templates

##### Technologies and frameworks
- Python 3.8+
- Docker / Milvus (vector database)
- Pandas for CSV/XLSX processing
- Jinja-style HTML templating
- SMTP/IMAP libraries for email I/O
- AWS Bedrock or OpenAI embeddings

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        direction TB
        Main["Main Orchestrator"]:::ingestion
        CSV["CSV Metadata Source"]:::ingestion
        XLSX["Excel KB Sheets"]:::ingestion
        DataDBProcessor["DataDB Processor"]:::ingestion
        VectorIngestion["Vector Ingestion Engine"]:::ingestion
    end

    %% Storage & Indexing Layer
    subgraph "Storage & Indexing"
        direction TB
        DockerCompose["Docker Compose (Milvus)"]:::storage
        Milvus["Milvus Vector Store"]:::storage
    end

    %% Query Engine Layer
    subgraph "Semantic Response Engine"
        direction TB
        EmbeddingModel["Embedding Model (OpenAI/AWS Bedrock)"]:::engine
        SemanticEngine["SemanticResponseEngine"]:::engine
        Templates["Jinja-Style HTML Templates"]:::engine
    end

    %% Email I/O Layer
    subgraph "Email I/O"
        direction TB
        EmailRetrieval["Email Retrieval Module"]:::email
        SemanticEngine -->|queries| Templates
        EmailResponder["Email Responder Module"]:::email
        ManualOverride["Manual Email Override Processor"]:::email
        SMTP["SMTP/IMAP Server"]:::external
    end

    %% Admin Utilities Layer
    subgraph "Admin Utilities & Tests"
        direction TB
        Config["Global Configuration"]:::admin
        ListCol["List Collections"]:::admin
        InspectCol["Inspect Collections"]:::admin
        DeleteCol["Delete All Collections"]:::admin
        TestIngest["Unit Test: test_ingest.py"]:::admin
        TestQuery["Unit Test: test_robust_query.py"]:::admin
    end

    %% Data Flow Connections
    CSV -->|ingest| DataDBProcessor
    XLSX -->|ingest| DataDBProcessor
    DataDBProcessor -->|vectorize| VectorIngestion
    VectorIngestion -->|store| Milvus
    DockerCompose -->|hosts| Milvus

    %% Query Flow
    EmailRetrieval -->|incoming ticket| SemanticEngine
    SemanticEngine -->|"generate response"| EmailResponder
    EmailResponder -->|send email via| SMTP
    ManualOverride -->|override response| EmailResponder

    %% Engine Interactions
    EmbeddingModel -->|embedding API| SemanticEngine
    Config -.-> DataDBProcessor
    Config -.-> VectorIngestion
    Config -.-> SemanticEngine
    Config -.-> EmailRetrieval
    Config -.-> EmailResponder

    %% Admin Tools
    ListCol -->|manage| Milvus
    InspectCol -->|manage| Milvus
    DeleteCol -->|manage| Milvus
    TestIngest -->|validates| VectorIngestion
    TestQuery -->|validates| SemanticEngine

    %% Orchestration
    Main --> DataDBProcessor
    Main --> VectorIngestion
    Main --> EmailRetrieval

    %% Click Events
    click Main "https://github.com/akshatv07/email-automation/blob/main/main.py"
    click CSV "https://github.com/akshatv07/email-automation/blob/main/data/datadb.csv"
    click XLSX "https://github.com/akshatv07/email-automation/tree/main/data/kb_sheets/"
    click DataDBProcessor "https://github.com/akshatv07/email-automation/blob/main/core/data_db_processor.py"
    click VectorIngestion "https://github.com/akshatv07/email-automation/blob/main/core/vector_ingestion_engine.py"
    click DockerCompose "https://github.com/akshatv07/email-automation/blob/main/docker-compose.yml"
    click EmbeddingModel "https://github.com/akshatv07/email-automation/blob/main/bedrock.py"
    click Config "https://github.com/akshatv07/email-automation/blob/main/config/settings.py"
    click Templates "https://github.com/akshatv07/email-automation/tree/main/templates/templates/"
    click EmailRetrieval "https://github.com/akshatv07/email-automation/blob/main/retrieve_email.py"
    click EmailResponder "https://github.com/akshatv07/email-automation/blob/main/email_responder.py"
    click ManualOverride "https://github.com/akshatv07/email-automation/blob/main/manual_email_processor.py"
    click ListCol "https://github.com/akshatv07/email-automation/blob/main/list_collections.py"
    click InspectCol "https://github.com/akshatv07/email-automation/blob/main/inspect_collections.py"
    click DeleteCol "https://github.com/akshatv07/email-automation/blob/main/delete_all_collections.py"
    click TestIngest "https://github.com/akshatv07/email-automation/blob/main/test_ingest.py"
    click TestQuery "https://github.com/akshatv07/email-automation/blob/main/test_robust_query.py"

    %% Styles
    classDef ingestion fill:#bbe1fa,stroke:#1a73e8
    classDef storage fill:#d0f4de,stroke:#34a853
    classDef engine fill:#fde2e4,stroke:#ea4335
    classDef email fill:#f3e5f5,stroke:#8e24aa
    classDef admin fill:#e0e0e0,stroke:#5f6368
    classDef external fill:#ffffff,stroke:#000000,stroke-dasharray:5 2
```
