# arjun618/dmp-25-tap-poc architecture

> Architecture diagram of the GitHub repository arjun618/dmp-25-tap-poc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arjun618/dmp-25-tap-poc
- Repository: https://github.com/arjun618/dmp-25-tap-poc

## Overview

##### Project Type and Purpose
- Type: Monolithic full-stack AI assistant (Proof-of-Concept)
- Purpose: Expose a Streamlit UI to answer natural language queries on student/teacher CSV data using a hybrid Retrieval-Augmented Generation (RAG) pipeline with FAISS vector search and a Large Language Model (local or cloud).

##### File Structure
- Top-level scripts:
– main.py: entry point that launches the Streamlit app
– ui.py: Streamlit UI components (frontend)
– query_processing.py: orchestrates RAG pipeline
– data_processing.py: handles structured data queries on pandas DataFrames
– vectorstore.py: manages FAISS index (index.faiss + index.pkl)
– llm.py: abstracts local or Hugging Face LLM usage
– config.py: runtime settings (e.g., USE_LOCAL_MODEL, HF token)
- Directories:
– data/: CSV datasets for students and teachers
– faiss_index/: serialized FAISS vector index
– models/ (optional): local GGUF model files
- Config & dependencies:
– .env, requirements.txt

##### Extract Additional Insights from README
- RAG pipeline:
1. User query → embedding → vector search (FAISS)
2. Retrieved documents + structured data analysis → LLM prompt
3. LLM generates response
- LLM options: local Llama-2 (CTransformers) or cloud Falcon-7B (Hugging Face API)
- UI shows queries, responses, and debug info

##### Main Components & Their Relationships
- Frontend (Streamlit UI – ui.py)
- Orchestrator (main.py + Streamlit runtime)
- Query Processor (query_processing.py)
– Calls Vector Store & Data Processing
- Vector Store (vectorstore.py + faiss_index files)
– Embedding model (SentenceTransformers)
- Structured Data Module (data_processing.py)
– Pandas DataFrame operations on CSVs
- LLM Module (llm.py)
– Local or remote inference
- Data Sources:
– CSV datasets
– FAISS index (pre-built)
- External Services:
– Hugging Face API (if USE_LOCAL_MODEL=False)

1. UI → Orchestrator
2. Orchestrator → Query Processor
3. Query Processor → Vector Store (retrieve top-k docs)
4. Query Processor → Data Processing (for structured queries)
5. Query Processor → LLM Module (generate answer)
6. LLM Module → Orchestrator → UI (response display)
7. Vector Store ↔ FAISS index files
8. LLM Module ↔ Local model files or Hugging Face API

##### Architectural Patterns & Design Principles
- Modular separation of concerns
- Pipeline pattern for RAG: Retrieval → Augmentation → Generation
- Config-driven behavior (local vs. cloud LLM)
- Data-driven design: CSV → embeddings → vector index → query

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Streamlit App"
        UI["Streamlit UI"]:::frontend
        ORCH["Orchestrator"]:::frontend
    end

    subgraph "Backend Modules"
        QP["Query Processor"]:::core
        VS["Vector Store"]:::core
        SD["Data Processing"]:::core
        LM["LLM Module"]:::core
    end

    subgraph "Data Stores"
        CSV1[(student_dataset.csv)]:::datastore
        CSV2[(teacher_dataset.csv)]:::datastore
        FAISS1[(index.faiss)]:::datastore
        FAISS2[(index.pkl)]:::datastore
        LOCAL_MODEL[(Local Model Files)]:::datastore
        CFG1[(config.py)]:::datastore
        CFG2[(.env)]:::datastore
        DEPS[(requirements.txt)]:::datastore
    end

    HF["Hugging Face API"]:::external

    UI -->|"user query"| ORCH
    ORCH -->|"invoke RAG"| QP
    QP -->|"embed→search"| VS
    VS ---|"reads/writes"| FAISS1
    VS ---|"reads/writes"| FAISS2
    QP -->|"structured query"| SD
    SD -->|"reads"| CSV1
    SD -->|"reads"| CSV2
    QP -->|"prompt→LLM"| LM
    LM -->|"response"| ORCH
    ORCH -->|"display"| UI
    LM ---|"local inference"| LOCAL_MODEL
    LM -->|"API call"| HF

    click UI "https://github.com/arjun618/dmp-25-tap-poc/blob/main/ui.py"
    click ORCH "https://github.com/arjun618/dmp-25-tap-poc/blob/main/main.py"
    click QP "https://github.com/arjun618/dmp-25-tap-poc/blob/main/query_processing.py"
    click VS "https://github.com/arjun618/dmp-25-tap-poc/blob/main/vectorstore.py"
    click FAISS1 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/faiss_index/index.faiss"
    click FAISS2 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/faiss_index/index.pkl"
    click SD "https://github.com/arjun618/dmp-25-tap-poc/blob/main/data_processing.py"
    click CSV1 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/data/student_dataset.csv"
    click CSV2 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/data/teacher_dataset.csv"
    click LM "https://github.com/arjun618/dmp-25-tap-poc/blob/main/llm.py"
    click CFG1 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/config.py"
    click CFG2 "https://github.com/arjun618/dmp-25-tap-poc/blob/main/.env"
    click DEPS "https://github.com/arjun618/dmp-25-tap-poc/blob/main/requirements.txt"

    classDef frontend fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef core fill:#E6FFE6,stroke:#2E8B57,color:#2E8B57;
    classDef datastore fill:#F0F0F0,stroke:#666666,color:#333333;
    classDef external fill:#FFF4C1,stroke:#DAA520,color:#B8860B;
```
