# timothyafolami/mse-ai architecture

> Architecture diagram of the GitHub repository timothyafolami/mse-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/timothyafolami/mse-ai
- Repository: https://github.com/timothyafolami/mse-ai

## Overview

##### Main Components
- User Interface (Frontend): Represented by the Streamlit application launched by main.py. This is where users upload files, trigger document processing, and submit queries.
- Document Upload & Temporary Storage: Include a component for handling file uploads, with references to folders like temp_uploads (as indicated in the README) and the data folder for sample data.
- Document Processing Pipeline:
- Data Loader Modules: Under src/data_loader there are separate modules for loading documents (doc_loader.py), handling PDFs (pdf_loader.py), and for processing unstructured data (unstructured_loader.py).
- Document Indexer: The doc_indexer.py module builds and manages vector indexes (stored under output/doc_indexes) using FAISS and serializes data via pickle (index.pkl).
- Settings Module: Represents configurable parameters for document processing.
- AI Functions / Query Generation: Located in src/ai_functions. Modules like prompt_functions.py and prompts.py are used to generate related queries or process natural language queries using an AI service.
- Database/Index Storage: Illustrate the vector database component (using FAISS) that underlies the document retrieval and advanced query creation.

##### Relationships & Data Flow
- Document Loader: Accepts the uploaded files and processes the content.
- Document Indexer: Receives processed text, creates, and updates a vector-based index stored in the output folder.
- Another branch should show the “AI Functions” component that takes user queries and interacts with external AI services (OpenAI/Groq through LangChain) to generate responses.
- That the “Document Index” (with FAISS) is a central repository used for efficient search and is stored on the filesystem (inside output/doc_indexes).

##### Architectural Patterns & Technologies
- Modular Architecture: Emphasize how the application’s source code is organized into distinct modules for AI functions and document data loading/processing.
- Service Separation: Show clear separation between the front-end (Streamlit view) and backend processing (data loader and indexer modules), as well as the external AI service integration.

##### Additional Recommendations
- Incorporate specific module names from the file tree (for example, show “pdf_loader.py” as part of the “Document Loader” block).
- Persistent storage paths for generated indices (output/doc_indexes) and reference that these are saved in the file system.
- Make annotations for any key settings or configurable items (e.g., API keys) to highlight where integrations are managed.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entities
    U("User"):::external

    %% Frontend Layer
    F("Streamlit UI"):::frontend

    %% Document Upload & Temporary Storage
    DTS("Document Upload & Temporary Storage"):::storage

    %% Backend - Document Processing Pipeline
    subgraph "Document Processing Pipeline"
        subgraph "Document Loader"
            DL("General Doc Loader"):::backend
            PL("PDF Loader"):::backend
            UL("Unstructured Data Loader"):::backend
        end
        DI("Document Indexer"):::backend
        PS("Persistent Vector Storage (FAISS Index)"):::storage
        SM("Settings Module"):::backend
    end

    %% AI Functions Layer
    subgraph "AI Functions / Query Generation"
        APF("AI Prompt Functions"):::backend
        AP("AI Prompts"):::backend
    end

    %% External Services
    EA("External AI Services (OpenAI/Groq)"):::external

    %% Data Flow Connections
    U -->|"initiates"| F
    F -->|"uploads files"| DTS
    DTS -->|"feeds documents"| DL
    DTS -->|"feeds documents"| PL
    DTS -->|"feeds documents"| UL
    DL -->|"processes content"| DI
    PL -->|"processes PDFs"| DI
    UL -->|"processes unstructured data"| DI
    DI -->|"builds & updates index"| PS
    SM -->|"configures"| DI
    F -->|"submits query"| APF
    APF -->|"generates query"| AP
    APF -->|"requests info"| DI
    DI -->|"provides vectors"| APF
    AP -->|"calls external API"| EA
    EA -->|"returns AI response"| AP
    AP -->|"sends result"| F

    %% Click Events
    click F "https://github.com/timothyafolami/mse-ai/blob/main/main.py"
    click DTS "https://github.com/timothyafolami/mse-ai/tree/main/data"
    click DL "https://github.com/timothyafolami/mse-ai/blob/main/src/data_loader/doc_loader.py"
    click PL "https://github.com/timothyafolami/mse-ai/blob/main/src/data_loader/pdf_loader.py"
    click UL "https://github.com/timothyafolami/mse-ai/blob/main/src/data_loader/unstructured_loader.py"
    click DI "https://github.com/timothyafolami/mse-ai/blob/main/src/data_loader/doc_indexer.py"
    click SM "https://github.com/timothyafolami/mse-ai/blob/main/src/data_loader/settings.py"
    click APF "https://github.com/timothyafolami/mse-ai/blob/main/src/ai_functions/prompt_functions.py"
    click AP "https://github.com/timothyafolami/mse-ai/blob/main/src/ai_functions/prompts.py"
    click PS "https://github.com/timothyafolami/mse-ai/tree/main/output/doc_indexes"

    %% Styles
    classDef frontend fill:#ffd699,stroke:#d35400,stroke-width:2px;
    classDef backend fill:#d6eaf8,stroke:#2874a6,stroke-width:2px;
    classDef storage fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef external fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
```
