How timothyafolami/mse-ai works
Main Components
• User Interface (Frontend): Represented by the Streamlit application launched by main.py. This is where users upload files, trigger document processing, and submit queries.
• Document Upload & Temporary Storage: Include a component for handling file uploads, with references to folders like temp_uploads (as indicated in the README) and the data folder for sample data.
• Document Processing Pipeline:
• Data Loader Modules: Under src/data_loader there are separate modules for loading documents (doc_loader.py), handling PDFs (pdf_loader.py), and for processing unstructured data (unstructured_loader.py).
• Document Indexer: The doc_indexer.py module builds and manages vector indexes (stored under output/doc_indexes) using FAISS and serializes data via pickle (index.pkl).
• Settings Module: Represents configurable parameters for document processing.
• AI Functions / Query Generation: Located in src/ai_functions. Modules like prompt_functions.py and prompts.py are used to generate related queries or process natural language queries using an AI service.
• Database/Index Storage: Illustrate the vector database component (using FAISS) that underlies the document retrieval and advanced query creation.
Relationships & Data Flow
• Document Loader: Accepts the uploaded files and processes the content.
• Document Indexer: Receives processed text, creates, and updates a vector-based index stored in the output folder.
• Another branch should show the “AI Functions” component that takes user queries and interacts with external AI services (OpenAI/Groq through LangChain) to generate responses.
• That the “Document Index” (with FAISS) is a central repository used for efficient search and is stored on the filesystem (inside output/doc_indexes).
Architectural Patterns & Technologies
• Modular Architecture: Emphasize how the application’s source code is organized into distinct modules for AI functions and document data loading/processing.
• Service Separation: Show clear separation between the front-end (Streamlit view) and backend processing (data loader and indexer modules), as well as the external AI service integration.
Additional Recommendations
• Incorporate specific module names from the file tree (for example, show “pdf_loader.py” as part of the “Document Loader” block).
• Persistent storage paths for generated indices (output/doc_indexes) and reference that these are saved in the file system.
• Make annotations for any key settings or configurable items (e.g., API keys) to highlight where integrations are managed.
Happy diagramming!