How arjun618/dmp-25-tap-poc works
Project Type and Purpose
• Type: Monolithic full-stack AI assistant (Proof-of-Concept)
• Purpose: Expose a Streamlit UI to answer natural language queries on student/teacher CSV data using a hybrid Retrieval-Augmented Generation (RAG) pipeline with FAISS vector search and a Large Language Model (local or cloud).
File Structure
• Top-level scripts:
– main.py: entry point that launches the Streamlit app
– ui.py: Streamlit UI components (frontend)
– query_processing.py: orchestrates RAG pipeline
– data_processing.py: handles structured data queries on pandas DataFrames
– vectorstore.py: manages FAISS index (index.faiss + index.pkl)
– llm.py: abstracts local or Hugging Face LLM usage
– config.py: runtime settings (e.g., USE_LOCAL_MODEL, HF token)
• Directories:
– data/: CSV datasets for students and teachers
– faiss_index/: serialized FAISS vector index
– models/ (optional): local GGUF model files
• Config & dependencies:
– .env, requirements.txt
Extract Additional Insights from README
• RAG pipeline:
1. User query → embedding → vector search (FAISS)
2. Retrieved documents + structured data analysis → LLM prompt
3. LLM generates response
• LLM options: local Llama-2 (CTransformers) or cloud Falcon-7B (Hugging Face API)
• UI shows queries, responses, and debug info
Main Components & Their Relationships
• Frontend (Streamlit UI – ui.py)
• Orchestrator (main.py + Streamlit runtime)
• Query Processor (query_processing.py)
– Calls Vector Store & Data Processing
• Vector Store (vectorstore.py + faiss_index files)
– Embedding model (SentenceTransformers)
• Structured Data Module (data_processing.py)
– Pandas DataFrame operations on CSVs
• LLM Module (llm.py)
– Local or remote inference
• Data Sources:
– CSV datasets
– FAISS index (pre-built)
• External Services:
– Hugging Face API (if USE_LOCAL_MODEL=False)
1. UI → Orchestrator
2. Orchestrator → Query Processor
3. Query Processor → Vector Store (retrieve top-k docs)
4. Query Processor → Data Processing (for structured queries)
5. Query Processor → LLM Module (generate answer)
6. LLM Module → Orchestrator → UI (response display)
7. Vector Store ↔ FAISS index files
8. LLM Module ↔ Local model files or Hugging Face API
Architectural Patterns & Design Principles
• Modular separation of concerns
• Pipeline pattern for RAG: Retrieval → Augmentation → Generation
• Config-driven behavior (local vs. cloud LLM)
• Data-driven design: CSV → embeddings → vector index → query