How anjanykumarjaiswal/visiontext-llm works
Overall System Purpose and Flow
• That the goal is to extract text and images from PDF files, embed them into a shared vector space, perform semantic retrieval via Pinecone, and feed the results to a multimodal LLM (Google Gemini) to generate answers.
• The application is exposed as a FastAPI service (see src/main.py) that handles query requests.
Main Components
• API Layer: Represented by the FastAPI server (src/main.py) which exposes endpoints (e.g., /run?query=...).
• PDF Processing Component: Represented by the preprocessing module (src/utils/preprocessing.py) which extracts text and images from PDF documents using libraries like PyPDF2, Fitz, and Pillow.
• Multimodal Embedding Component: Found in src/models/embeddings.py, where the MultiModalEmbedder utilizes a Hugging Face ‘AlignModel’ to convert both texts and images into embeddings.
• Vector Database Interface: The Pinecone interaction functionalities found in src/utils/pineconedb.py – handling upserts and querying the vector store.
• Answer Generation Component: In src/models/gemini.py which holds the GenerateGeminiResponse class to process the retrieved embeddings/text/images and generate a multimodal answer using the Google Gemini API.
Relationships and Data Flow
• The query is processed and, if needed, triggers the PDF Processing component to extract content.
• The extracted content is passed to the Multimodal Embedding component, which generates embeddings from text and images.
• The embeddings are then upserted into or queried from Pinecone via the Database Interface.
• Finally, the retrieved relevant content is sent to the Answer Generation component, where the multimodal LLM (Google Gemini) formulates a response.
• Return the answer to the API layer and hence to the user.
Architectural Patterns / Design Principles
• Modular Design: The separation into distinct modules (API, preprocessing, embedding, database interaction, and answer generation) follows a clear separation of concerns.
• Pipeline Architecture: The sequential flow from document processing to embedding to retrieval, and finally, language model-based response generation.
• Use of External Services: Clearly delineate integration with external services (Hugging Face, Pinecone, Google Gemini).
Specify Technologies and Frameworks
• API: FastAPI powered by Uvicorn.
• Embedding: Hugging Face Transformers with the AlignModel.
• Vector Database: Pinecone.
• LLM: Google Gemini.
• PDF / Image Processing: PyPDF2, Fitz, and Pillow.
• Containerization: Docker (as hinted by Dockerfile and .dockerignore) suggests the possibility of containerized deployment.