# starpig1129/datagen architecture

> Architecture diagram of the GitHub repository starpig1129/datagen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/starpig1129/datagen
- Repository: https://github.com/starpig1129/datagen

## Overview

##### Project Overview and Main Purpose
- DATAGEN is a multi-agent system built for automated data analysis, research, visualization, and report generation.
- The platform leverages cutting-edge AI components (e.g., LangChain, OpenAI’s GPT models, LangGraph) and consists of specialized agents that collaborate on various tasks from hypothesis generation to quality review.
- The system is designed to be both scalable and production-ready with robust memory and workflow management.

##### And Separate the Core Architectural Components
a. Multi-Agent Suite (Agent Components):
- Specialized agent from the “agent” directory as separate components:
- hypothesis_agent: Generates research hypotheses.
- process_agent: Oversees and coordinates the overall research process.
- visualization_agent: Handles data visualization and graphical reporting.
- code_agent: Writes code for data analysis.
- search_agent: Performs literature/web searches.
- report_agent: Compiles the research reports.
- quality_review_agent: Conducts quality assurance tasks.
- note_agent: Manages state and retains context (smart memory management).
- refiner_agent: Refines and improves on preliminary outputs.

b. Core Framework and Workflow Management (Core Components):
- From the “core” directory:
- language_models.py: Likely abstracts interactions with external language model APIs services.
- node.py and router.py: Indicate modular node-based processing and routing within the agent coordination flow.
- state.py: Manages global state and context propagation.
- workflow.py: Defines the processing pipeline using LangGraph to create the state graph of the workflow.

c. Supporting Tools and Utilities (Tools Directory & Others):
- Within the “tools” directory, components like FileEdit.py, internet.py, and basetool.py provide additional utilities (e.g., file manipulation, internet queries) that support the agents.
- As auxiliary components supplementing the main pipeline.

d. Peripheral/Entry Components:
- main.py and main.ipynb act as entry points for running the system.
- create_agent.py, load_cfg.py, and logger.py represent initial configuration, agent instantiation, and logging functionality.

##### Relationships and Data/Control Flow
- Flow that begins with system initialization (via main.py or the Jupyter Notebook):
- Initial configuration is loaded (load_cfg.py and .env setup) and agents are instantiated (create_agent.py).
- The orchestration engine (via the core module) then dispatches tasks, sending input to the relevant agents.
- If applicable, highlight feedback loops, such as:
- The note_agent’s state updates feeding back into the core’s workflow management.
- Quality review (quality_review_agent) that can trigger further refinements by the refiner_agent.

##### Architectural Patterns and Technologies
- Separation of concerns: Clear division between the agent logic, core orchestration, and utility tools.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Bootstrap / Initialization
    subgraph "Initialization"
        EP1("main.py"):::bootstrap
        EP2("main.ipynb"):::bootstrap
        CFG("load_cfg.py"):::bootstrap
        AGINS("create_agent.py"):::bootstrap
        LOG("logger.py"):::bootstrap
    end

    %% Core Orchestration Components
    subgraph "Core Orchestration"
        LM("Language Models"):::core
        NP("Node Processing"):::core
        RT("Router"):::core
        GS("Global State"):::core
        WF("Workflow"):::core
    end

    %% Agent Suite
    subgraph "Agent Suite"
        HA("Hypothesis Agent"):::agent
        PA("Process Agent"):::agent
        VA("Visualization Agent"):::agent
        CA("Code Agent"):::agent
        SA("Search Agent"):::agent
        RA("Report Agent"):::agent
        QRA("Quality Review Agent"):::agent
        NA("Note Agent"):::agent
        RFA("Refiner Agent"):::agent
    end

    %% Utility Tools
    subgraph "Utility Tools"
        FE("FileEdit"):::tool
        BT("Basetool"):::tool
        IN("Internet"):::tool
    end

    %% External Services
    subgraph "External Services"
        EXT("External AI Services"):::external
    end

    %% Bootstrap Flow
    EP1 --> CFG
    EP2 --> CFG
    CFG --> AGINS
    AGINS --> LOG
    LOG --> WF

    %% Core internal flow
    LM --> NP
    NP --> RT
    RT --> GS
    GS --> WF
    LM --> EXT

    %% Core to Agent Suite
    WF --> HA
    WF --> PA
    WF --> VA
    WF --> CA
    WF --> SA
    WF --> RA
    WF --> QRA
    WF --> NA
    WF --> RFA

    %% Agent intercommunication (feedback)
    QRA --> RFA
    NA --> GS

    %% Utility Tools supporting Workflow
    FE --> WF
    BT --> WF
    IN --> WF

    %% Click Events
    click EP1 "https://github.com/starpig1129/datagen/blob/main/main.py"
    click EP2 "https://github.com/starpig1129/datagen/blob/main/main.ipynb"
    click CFG "https://github.com/starpig1129/datagen/blob/main/load_cfg.py"
    click AGINS "https://github.com/starpig1129/datagen/blob/main/create_agent.py"
    click LOG "https://github.com/starpig1129/datagen/blob/main/logger.py"
    click LM "https://github.com/starpig1129/datagen/blob/main/core/language_models.py"
    click NP "https://github.com/starpig1129/datagen/blob/main/core/node.py"
    click RT "https://github.com/starpig1129/datagen/blob/main/core/router.py"
    click GS "https://github.com/starpig1129/datagen/blob/main/core/state.py"
    click WF "https://github.com/starpig1129/datagen/blob/main/core/workflow.py"
    click HA "https://github.com/starpig1129/datagen/blob/main/agent/hypothesis_agent.py"
    click PA "https://github.com/starpig1129/datagen/blob/main/agent/process_agent.py"
    click VA "https://github.com/starpig1129/datagen/blob/main/agent/visualization_agent.py"
    click CA "https://github.com/starpig1129/datagen/blob/main/agent/code_agent.py"
    click SA "https://github.com/starpig1129/datagen/blob/main/agent/search_agent.py"
    click RA "https://github.com/starpig1129/datagen/blob/main/agent/report_agent.py"
    click QRA "https://github.com/starpig1129/datagen/blob/main/agent/quality_review_agent.py"
    click NA "https://github.com/starpig1129/datagen/blob/main/agent/note_agent.py"
    click RFA "https://github.com/starpig1129/datagen/blob/main/agent/refiner_agent.py"
    click FE "https://github.com/starpig1129/datagen/blob/main/tools/FileEdit.py"
    click BT "https://github.com/starpig1129/datagen/blob/main/tools/basetool.py"
    click IN "https://github.com/starpig1129/datagen/blob/main/tools/internet.py"

    %% Styles
    classDef bootstrap fill:#ffeb3b,stroke:#f57f17,stroke-width:2px;
    classDef core fill:#8bc34a,stroke:#558b2f,stroke-width:2px;
    classDef agent fill:#03a9f4,stroke:#0277bd,stroke-width:2px;
    classDef tool fill:#9c27b0,stroke:#6a1b9a,stroke-width:2px;
    classDef external fill:#ff9800,stroke:#ef6c00,stroke-width:2px;
```
