How nilesh666/involyticsn works
Project type and purpose
• This is an AI-driven invoice analytics pipeline and agentic workflow. It ingests invoice images, uses a vision-language model (VLM) plus Pydantic schemas to extract structured data, stores raw results in MongoDB, runs ETL via Apache Airflow into Postgres, and exposes two agentic query interfaces (invoice-specific and analytics) with observability via Langfuse.
File structure
• Top-level directories:
• airflow/ – Dockerfile, DAGs (mongo_to_postgres), tests, configs, packages.txt, requirements.txt
• core/ – agent/tool definitions (agents.py, tools.py), processing logic, Pydantic schemas
• mongo/ – file and operations helpers for raw data capture
• utils/ – config loading, logging, custom exceptions
• notebook_testing/ – experimental Jupyter notebooks
• example_data_output/ – sample payloads and response JSON
• Root files: setup.py, requirements.txt, example_env.txt indicate a Python package installable via pip.
• Key patterns: ETL DAGs under airflow/dags, modular agent definitions in core/, separation of storage layers (mongo/ vs Postgres via Airflow).
Extract additional insights from README
• Two-agent system: Invoice Agent (reads from MongoDB) and Analytics Agent (reads from Postgres)
• Workflow: Image upload → VLM extraction → MongoDB raw store → Airflow ETL → Postgres processed store → Agents serve user queries
• Observability: Langfuse integrates at model/agent call boundaries
1. Client / Uploader (API or UI placeholder)
2. Vision-Language Model (Hugging Face Qwen) + Pydantic Validator
3. Raw Data Store (MongoDB)
4. ETL Orchestrator (Apache Airflow DAGs)
5. Processed Data Store (Postgres)
6. Agent Framework (Agno Agents)
– Invoice Agent
– Analytics Agent
7. Observability Layer (Langfuse)
8. Optional: Notebook/CLI for testing and exploration
b. Relationships & data flow
• Observability hooks around VLM calls and agent actions
c. Architectural patterns & principles
• Clear separation of concerns: ingestion, raw capture, ETL, query serving
• Agent-based micro-services: two specialized agents
• Schema validation at the model boundary (Pydantic)
• Modular DAGs for maintainable workflow
• Pluggable observability
d. Key technologies & frameworks
• Python / Pydantic
• Hugging Face Qwen/VLM
• Agno Agents
• MongoDB & PyMongo helpers (mongo/operations.py)
• Apache Airflow (DAG in airflow/dags/mongo_to_postgres.py)
• Postgres
• Langfuse (observability)
• Docker (airflow/Dockerfile)