# leorigasaki/insightai architecture

> Architecture diagram of the GitHub repository leorigasaki/insightai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leorigasaki/insightai
- Repository: https://github.com/leorigasaki/insightai

## Overview

##### Project Type and Purpose
- InsightAI is an open-source Python library for conversational data analysis using Large Language Models (LLMs).
- It provides a natural-language interface over pandas DataFrames or SQLite databases, orchestrating multiple specialized AI agents to generate code, analyze data, create visualizations, and produce reports.

##### High-Level Architectural Components
a. Client Layer
- Your Python script or REPL calls `InsightAI(...)` and invokes `pd_agent_converse(...)`.
b. Core Orchestrator (`insightai/insightai.py`)
- Initializes configuration, credentials, I/O managers, and logging.
- Acts as the central “Agent Manager,” routing tasks to specialized agents.
c. Agent Modules
- ExpertSelector (chooses the right agent based on the user query and context)
- DataAnalyst (statistical analysis & visualizations)
- SQLAnalyst (database queries via SQLite)
- DataCleaningExpert (data quality checks & fixes)
- CodeGenerator (generates Python code for analysis/plots)
- ErrorCorrector (debugs and repairs code execution errors)
- ReportGenerator (compiles results into markdown or PDF reports)
d. Model Adapters
- openai_models.py (wraps OpenAI API calls)
- groq_models.py (wraps Groq inference calls)
- gemini_models.py (wraps Gemini APIs)
e. Supporting Libraries
- prompts.py (LLM prompt templates)
- df_ontology.py (data schema/metadata extraction)
- func_calls.py, utils.py, reg_ex.py (helper functions)
- output_manager.py (writes charts, code, reports to disk)
- log_manager.py (captures chain_id, tokens used, cost, duration)
f. Data Layer
- Pandas DataFrame or SQLite DB (via db_path)
- `models.json` for model configurations
- Optional JSON for prompts and LLM_CONFIG
g. Visualization & Reporting
- Charts saved under `visualization/` folder
- Generated markdown/PDF via ReportGenerator

##### Component Interactions and Data Flow
1. User code → Core Orchestrator → ExpertSelector
2. ExpertSelector dispatches to one or more agents
- Each agent: retrieves context (DataFrame or DB schema), builds prompt via prompts.py, calls model adapter (OpenAI/Groq/Gemini).
- Agents stream responses (real-time processing) back to Orchestrator.
3. CodeGenerator produces Python snippets → Orchestrator executes in sandbox → ErrorCorrector may intervene
4. DataAnalyst/SQLAnalyst executes on DF/DB → OutputManager saves visualizations
5. ReportGenerator collates outputs (logs, charts, code, summaries) → writes final report
6. LogManager records every LLM call with metadata to `insightai_consolidated_log.json`

##### Architectural Patterns and Principles
- Agent-Based Micro-Orchestration: loosely coupled AI “agents” each with single responsibility.
- Strategy Pattern: ExpertSelector picks the right strategy/agent.
- Adapter Pattern: model wrappers abstract differences between OpenAI, Groq, and Gemini APIs.
- Pipeline/Pipeline with Error-Recovery: Code generation → execution → error catching & correction.
- Modular and Extendable: drop-in new model adapters, prompts, or agents.

##### Detailed Elements to Capture
- Labels for each agent and adapter file/module.
- Prompt flow: prompts.py → model adapter → LLM.
- Code execution sandbox and error correction loop.
- Output artifacts: charts in `visualization/`, markdown/PDF reports, logs file.
- Configuration sources: env vars, `models.json`, JSON templates.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration & Logging" 
        direction TB
        EnvVars["Environment Variables"]:::config
        ModelsConfig["models.json"]:::config
        OptionalJSON["Optional JSON Templates"]:::config
        LogManager["LogManager"]:::config
        LogManager -->|writes logs to| LogFile["insightai_consolidated_log.json"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        PandasDF["Pandas DataFrame"]:::data
        SQLiteDB["SQLite Database (*.db)"]:::data
    end

    %% Core Library
    subgraph "Core Library (insightai/)"
        direction TB
        Orchestrator["Core Orchestrator\n(insightai.py)"]:::core
        ExpertSelector["ExpertSelector Agent"]:::core
        DataAnalyst["DataAnalyst Agent"]:::core
        SQLAnalyst["SQLAnalyst Agent"]:::core
        DataCleaning["DataCleaningExpert Agent"]:::core
        CodeGen["CodeGenerator Agent"]:::core
        ErrorCorrector["ErrorCorrector Agent"]:::core
        ReportGen["ReportGenerator Agent"]:::core

        Prompts["prompts.py"]:::helper
        Ontology["df_ontology.py"]:::helper
        FuncCalls["func_calls.py"]:::helper
        Utils["utils.py"]:::helper
        RegEx["reg_ex.py"]:::helper

        OutputMgr["output_manager.py"]:::helper

        OpenAIAdapter["openai_models.py"]:::adapter
        GroqAdapter["groq_models.py"]:::adapter
        GeminiAdapter["gemini_models.py"]:::adapter
    end

    %% External Services
    subgraph "External LLM Services"
        direction TB
        OpenAIAPI["OpenAI API"]:::external
        GroqAPI["Groq API"]:::external
        GeminiAPI["Gemini API"]:::external
    end

    %% Artifacts
    subgraph "Artifacts"
        direction TB
        VizDiagrams["Mermaid diagrams\n(visualization/*.mmd)"]:::artifact
        Charts["Charts (visualization/)"]:::artifact
        ReportMD["Report Markdown\n(data_analysis_report_1741536296.md)"]:::artifact
    end

    %% User Interaction
    User["User Script / REPL"]:::external
    User -->|invokes pd_agent_converse| Orchestrator
    Orchestrator -->|selects expert| ExpertSelector

    %% Expert Dispatch
    ExpertSelector --> DataAnalyst
    ExpertSelector --> SQLAnalyst
    ExpertSelector --> DataCleaning
    ExpertSelector --> CodeGen
    ExpertSelector --> ErrorCorrector
    ExpertSelector --> ReportGen

    %% Agents use helpers
    DataAnalyst -->|uses| Ontology
    SQLAnalyst -->|uses| Ontology
    DataCleaning -->|uses| FuncCalls
    CodeGen -->|uses| Prompts
    ReportGen -->|uses| Prompts

    ExpertSelector -->|builds prompts via| Prompts

    %% Modeling adapters
    Prompts --> OpenAIAdapter
    Prompts --> GroqAdapter
    Prompts --> GeminiAdapter

    DataAnalyst -->|calls| OpenAIAdapter
    DataAnalyst -->|calls| GroqAdapter
    DataAnalyst -->|calls| GeminiAdapter

    SQLAnalyst -->|calls| OpenAIAdapter
    SQLAnalyst -->|calls| GroqAdapter
    SQLAnalyst -->|calls| GeminiAdapter

    DataCleaning -->|calls| OpenAIAdapter
    CodeGen -->|calls| OpenAIAdapter
    ErrorCorrector -->|calls| OpenAIAdapter
    ReportGen -->|calls| OpenAIAdapter

    OpenAIAdapter -->|streams response| Orchestrator
    GroqAdapter -->|streams response| Orchestrator
    GeminiAdapter -->|streams response| Orchestrator

    OpenAIAdapter --> OpenAIAPI
    GroqAdapter --> GroqAPI
    GeminiAdapter --> GeminiAPI

    %% Data sources usage
    DataAnalyst -->|reads/writes| PandasDF
    SQLAnalyst -->|queries| SQLiteDB

    %% Code execution and error correction
    CodeGen --> Sandbox["Execution Sandbox"]:::core
    Sandbox -->|error| ErrorCorrector
    ErrorCorrector --> Sandbox

    %% Output management
    DataAnalyst -->|produces| OutputMgr
    SQLAnalyst -->|produces| OutputMgr
    CodeGen -->|produces| OutputMgr

    OutputMgr --> Charts
    OutputMgr --> VizDiagrams

    ReportGen --> Charts
    ReportGen --> ReportMD

    %% Logging
    Orchestrator --> LogManager

    %% Click events
    click Orchestrator "https://github.com/leorigasaki/insightai/blob/main/insightai/insightai.py"
    click OpenAIAdapter "https://github.com/leorigasaki/insightai/blob/main/insightai/openai_models.py"
    click GroqAdapter "https://github.com/leorigasaki/insightai/blob/main/insightai/groq_models.py"
    click GeminiAdapter "https://github.com/leorigasaki/insightai/blob/main/insightai/gemini_models.py"
    click Prompts "https://github.com/leorigasaki/insightai/blob/main/insightai/prompts.py"
    click Ontology "https://github.com/leorigasaki/insightai/blob/main/insightai/df_ontology.py"
    click FuncCalls "https://github.com/leorigasaki/insightai/blob/main/insightai/func_calls.py"
    click Utils "https://github.com/leorigasaki/insightai/blob/main/insightai/utils.py"
    click RegEx "https://github.com/leorigasaki/insightai/blob/main/insightai/reg_ex.py"
    click OutputMgr "https://github.com/leorigasaki/insightai/blob/main/insightai/output_manager.py"
    click LogManager "https://github.com/leorigasaki/insightai/blob/main/insightai/log_manager.py"
    click ModelsConfig "https://github.com/leorigasaki/insightai/blob/main/models.json"
    click SQLiteDB "https://github.com/leorigasaki/insightai/tree/main/examples/database/"
    click VizDiagrams "https://github.com/leorigasaki/insightai/tree/main/visualization/"
    click ReportMD "https://github.com/leorigasaki/insightai/blob/main/data_analysis_report_1741536296.md"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef adapter fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef helper fill:#87CEFA,stroke:#000,stroke-width:1px;
    classDef data fill:#98FB98,stroke:#000,stroke-width:1px;
    classDef config fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef external fill:#FFA07A,stroke:#000,stroke-width:1px;
    classDef artifact fill:#DDA0DD,stroke:#000,stroke-width:1px;
```
