How leorigasaki/insightai works
Project Type and Purpose
• InsightAI is an open-source Python library for conversational data analysis using Large Language Models (LLMs).
• It provides a natural-language interface over pandas DataFrames or SQLite databases, orchestrating multiple specialized AI agents to generate code, analyze data, create visualizations, and produce reports.
High-Level Architectural Components
a. Client Layer
• Your Python script or REPL calls InsightAI(...) and invokes pd_agent_converse(...).
b. Core Orchestrator (insightai/insightai.py)
• Initializes configuration, credentials, I/O managers, and logging.
• Acts as the central “Agent Manager,” routing tasks to specialized agents.
c. Agent Modules
• ExpertSelector (chooses the right agent based on the user query and context)
• DataAnalyst (statistical analysis & visualizations)
• SQLAnalyst (database queries via SQLite)
• DataCleaningExpert (data quality checks & fixes)
• CodeGenerator (generates Python code for analysis/plots)
• ErrorCorrector (debugs and repairs code execution errors)
• ReportGenerator (compiles results into markdown or PDF reports)
d. Model Adapters
• openai_models.py (wraps OpenAI API calls)
• groq_models.py (wraps Groq inference calls)
• gemini_models.py (wraps Gemini APIs)
e. Supporting Libraries
• prompts.py (LLM prompt templates)
• df_ontology.py (data schema/metadata extraction)
• func_calls.py, utils.py, reg_ex.py (helper functions)
• output_manager.py (writes charts, code, reports to disk)
• log_manager.py (captures chain_id, tokens used, cost, duration)
f. Data Layer
• Pandas DataFrame or SQLite DB (via db_path)
• models.json for model configurations
• Optional JSON for prompts and LLM_CONFIG
g. Visualization & Reporting
• Charts saved under visualization/ folder
• Generated markdown/PDF via ReportGenerator
Component Interactions and Data Flow
1. User code → Core Orchestrator → ExpertSelector
2. ExpertSelector dispatches to one or more agents
• Each agent: retrieves context (DataFrame or DB schema), builds prompt via prompts.py, calls model adapter (OpenAI/Groq/Gemini).
• Agents stream responses (real-time processing) back to Orchestrator.
3. CodeGenerator produces Python snippets → Orchestrator executes in sandbox → ErrorCorrector may intervene
4. DataAnalyst/SQLAnalyst executes on DF/DB → OutputManager saves visualizations
5. ReportGenerator collates outputs (logs, charts, code, summaries) → writes final report
6. LogManager records every LLM call with metadata to insightai_consolidated_log.json
Architectural Patterns and Principles
• Agent-Based Micro-Orchestration: loosely coupled AI “agents” each with single responsibility.
• Strategy Pattern: ExpertSelector picks the right strategy/agent.
• Adapter Pattern: model wrappers abstract differences between OpenAI, Groq, and Gemini APIs.
• Pipeline/Pipeline with Error-Recovery: Code generation → execution → error catching & correction.
• Modular and Extendable: drop-in new model adapters, prompts, or agents.
Detailed Elements to Capture
• Labels for each agent and adapter file/module.
• Prompt flow: prompts.py → model adapter → LLM.
• Code execution sandbox and error correction loop.
• Output artifacts: charts in visualization/, markdown/PDF reports, logs file.
• Configuration sources: env vars, models.json, JSON templates.