How izzhamburhan/agentai-my works
Project Type and Purpose
• The project is a conversational agent that processes PDFs and CSV files (e.g., world population statistics) and supports taking notes. It employs a Retrieval-Augmented Generation (RAG) strategy, leveraging vector stores (found in directories like “combined_index” and “malaysia”) that hold precomputed indices, a document store, graph store, and image vector store.
• It is designed to receive file inputs, compute embeddings and indexes behind the scenes, and then interact with the user through an interface (using Streamlit as indicated in the README instructions).
File Structure Insights
• Top-level scripts: main.py, pdf.py, note_engine.py, and prompts.py. These implement core logic for document ingestion (pdf.py), note handling (note_engine.py), and agent prompt management.
• Data folders: “data” which contains sample PDFs, text, and CSV files. This folder serves as the raw input for the system.
• Index directories: “combined_index” and “malaysia” contain JSON files like default__vector_store.json, docstore.json, graph_store.json, etc., which organize the preprocessed document, vector, and graph data – essential for the RAG functionality.
• Configuration files: requirements.txt ensures that the dependencies (e.g., OpenAI API, Streamlit, vector store libraries) are clearly defined.
Components and Their Interactions
• User Interface (Frontend):
• Based on Streamlit (run via main.py), this is the interactive entry point where the user uploads PDFs and interacts with the conversational agent.
• Conversational Agent Core:
• Uses the ReAct framework with GPT-3.5 for dynamic responses.
• Processes questions, interprets user prompts, and triggers workflows (via prompts.py for instruction management).
• Document Processing Module:
• Handled by pdf.py for parsing incoming PDFs.
• Ingests documents and passes text to indexing processes.
• Engine:
• Managed via note_engine.py, where the system processes and stores user notes.
• Knowledge Base / Retrieval Module:
• Built around the vector store and associated index files (in “combined_index” and “malaysia”). These stores include document and graph representations to quickly retrieve relevant data to answer user queries.
• Acts as the RAG backbone: extracting document segments, verifying relevance, and feeding context into GPT-3.5.
• External Services:
• The integration with OpenAI’s API for GPT-3.5 is crucial.
Architectural Patterns and Design Principles
• Separation of concerns: The project splits functionality into distinct modules (UI, document ingestion, note-taking, and knowledge retrieval).
• Modular design: Each Python script (main.py, pdf.py, note_engine.py, prompts.py) acts as a module responsible for a particular aspect of the system.
• Data flow driven by triggers from user interaction that initiate document processing, vector indexing, and natural language processing queries.
• Use of external APIs (OpenAI GPT-3.5) and local vector store storage allows for a loosely coupled, scalable system.