# nilesh666/involyticsn architecture

> Architecture diagram of the GitHub repository nilesh666/involyticsn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nilesh666/involyticsn
- Repository: https://github.com/nilesh666/involyticsn

## Overview

##### Project type and purpose
- This is an AI-driven invoice analytics pipeline and agentic workflow. It ingests invoice images, uses a vision-language model (VLM) plus Pydantic schemas to extract structured data, stores raw results in MongoDB, runs ETL via Apache Airflow into Postgres, and exposes two agentic query interfaces (invoice-specific and analytics) with observability via Langfuse.

##### File structure
- Top-level directories:
- airflow/ – Dockerfile, DAGs (mongo_to_postgres), tests, configs, packages.txt, requirements.txt
- core/ – agent/tool definitions (agents.py, tools.py), processing logic, Pydantic schemas
- mongo/ – file and operations helpers for raw data capture
- utils/ – config loading, logging, custom exceptions
- notebook_testing/ – experimental Jupyter notebooks
- example_data_output/ – sample payloads and response JSON
- Root files: setup.py, requirements.txt, example_env.txt indicate a Python package installable via pip.
- Key patterns: ETL DAGs under airflow/dags, modular agent definitions in core/, separation of storage layers (mongo/ vs Postgres via Airflow).

##### Extract additional insights from README
- Two-agent system: Invoice Agent (reads from MongoDB) and Analytics Agent (reads from Postgres)
- Workflow: Image upload → VLM extraction → MongoDB raw store → Airflow ETL → Postgres processed store → Agents serve user queries
- Observability: Langfuse integrates at model/agent call boundaries

1. Client / Uploader (API or UI placeholder)
2. Vision-Language Model (Hugging Face Qwen) + Pydantic Validator
3. Raw Data Store (MongoDB)
4. ETL Orchestrator (Apache Airflow DAGs)
5. Processed Data Store (Postgres)
6. Agent Framework (Agno Agents)
– Invoice Agent
– Analytics Agent
7. Observability Layer (Langfuse)
8. Optional: Notebook/CLI for testing and exploration
b. Relationships & data flow
- Observability hooks around VLM calls and agent actions
c. Architectural patterns & principles
- Clear separation of concerns: ingestion, raw capture, ETL, query serving
- Agent-based micro-services: two specialized agents
- Schema validation at the model boundary (Pydantic)
- Modular DAGs for maintainable workflow
- Pluggable observability
d. Key technologies & frameworks
- Python / Pydantic
- Hugging Face Qwen/VLM
- Agno Agents
- MongoDB & PyMongo helpers (mongo/operations.py)
- Apache Airflow (DAG in airflow/dags/mongo_to_postgres.py)
- Postgres
- Langfuse (observability)
- Docker (airflow/Dockerfile)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    subgraph "Ingestion"
        Client["Client / Uploader"]:::external
        VLM["VLM + Pydantic Validator"]:::compute
    end
    Client -->|"invoice image"| VLM
    click VLM "https://github.com/nilesh666/involyticsn/blob/main/core/processing.py"

    %% Raw Data Store
    subgraph "Raw Storage"
        Mongo["MongoDB Raw Store"]:::datastore
    end
    VLM -->|"raw JSON invoice"| Mongo
    click Mongo "https://github.com/nilesh666/involyticsn/blob/main/mongo/operations.py"
    click Mongo "https://github.com/nilesh666/involyticsn/blob/main/mongo/files.py"

    %% ETL Orchestrator Layer
    subgraph "ETL Orchestrator"
        AirflowDocker["Airflow Docker"]:::compute
        AirflowConfig["Airflow Config & Deps"]:::compute
        AirflowDAG["Airflow DAG (mongo_to_postgres.py)"]:::compute
    end
    click AirflowDocker "https://github.com/nilesh666/involyticsn/tree/main/airflow/Dockerfile"
    click AirflowConfig "https://github.com/nilesh666/involyticsn/blob/main/airflow/packages.txt"
    click AirflowConfig "https://github.com/nilesh666/involyticsn/blob/main/airflow/requirements.txt"
    click AirflowDAG "https://github.com/nilesh666/involyticsn/blob/main/airflow/dags/mongo_to_postgres.py"
    Mongo -->|"raw JSON invoice"| AirflowDAG
    AirflowDAG -->|"transformed rows"| Postgres

    %% Processed Data Store
    subgraph "Processed Storage"
        Postgres["Postgres Processed Store"]:::datastore
    end

    %% Agent Framework Layer
    subgraph "Agent Framework"
        AgentsModule["Agent Framework"]:::compute
        InvoiceAgent["Invoice Agent"]:::compute
        AnalyticsAgent["Analytics Agent"]:::compute
    end
    click AgentsModule "https://github.com/nilesh666/involyticsn/blob/main/core/agents.py"
    InvoiceAgent -->|"queries raw data"| Mongo
    AnalyticsAgent -->|"queries processed data"| Postgres

    %% Observability Layer
    subgraph "Observability"
        Langfuse["Langfuse"]:::external
    end
    VLM -.->|"model call logs"| Langfuse
    InvoiceAgent -.->|"agent actions"| Langfuse
    AnalyticsAgent -.->|"agent actions"| Langfuse
    click Langfuse "https://github.com/nilesh666/involyticsn/blob/main/utils/logger.py"

    %% Utilities
    subgraph "Utilities"
        Config["Config Loader"]:::compute
        Logger["Logging & Observability Hooks"]:::compute
        Exceptions["Custom Exceptions"]:::compute
    end
    click Config "https://github.com/nilesh666/involyticsn/blob/main/utils/config.py"
    click Logger "https://github.com/nilesh666/involyticsn/blob/main/utils/logger.py"
    click Exceptions "https://github.com/nilesh666/involyticsn/blob/main/utils/custom_exception.py"

    %% Testing & Sample Data
    subgraph "Testing & Samples"
        Notebook["Agent Testing Notebook"]:::external
        Samples["Sample I/O Payloads"]:::external
    end
    click Notebook "https://github.com/nilesh666/involyticsn/blob/main/notebook_testing/Agno_agents.ipynb"
    click Samples "https://github.com/nilesh666/involyticsn/blob/main/example_data_output/data_to_load.json"
    click Samples "https://github.com/nilesh666/involyticsn/blob/main/example_data_output/vlm__response_example.json"

    %% Legend
    classDef compute fill:#BBDEFB,stroke:#1E88E5,stroke-width:1px
    classDef datastore fill:#FFE0B2,stroke:#FB8C00,stroke-width:1px
    classDef external fill:#C8E6C9,stroke:#43A047,stroke-width:1px
```
