How softlandia-ltd/vision-is-all-you-need works
Data Flow Representation
a) Document Processing Flow:
PDF Upload → PDF to Image Conversion → VLM Processing → Vector Storage
b) Query Flow:
User Query → Query Embedding → Vector Search → Image+Query Processing → Response Generation
Specific Component Details
• File Upload Interface
• Question Interface
• Attention Map Visualization
• API Client Module
• PDF Processing Service (pdf_to_image.py)
• ColPali VLM Service (colpali.py)
• Vector Database Client (qdrant_client.py)
• Main RAG Logic (vrag.py)
• OpenAI GPT-4V
• Hugging Face Transformers
• QDrant Vector DB
Important Technical Details
Annotations showing:
• TypeScript/React for frontend
• Python 3.11+ for backend
• Modal deployment platform
• Vector embedding process
• PDF to image conversion using pypdfium
• ColPali VLM implementation
• GPT-4V/GPT-4V-mini integration
Architecture Patterns to Highlight
• RESTful API communication between frontend and backend
• Asynchronous processing for PDF handling
• Vector similarity search patterns
• Vision-language model pipeline
Additional Notes
• Version information for key components
• Scaling considerations
• Security implementations (API keys, authentication)
• Development vs. production configurations
Remember to maintain a balance between high-level architecture and important implementation details, as this system has unique characteristics in its vision-based RAG approach that should be clearly represented.