# micic-mihajlo/datalore architecture

> Architecture diagram of the GitHub repository micic-mihajlo/datalore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/micic-mihajlo/datalore
- Repository: https://github.com/micic-mihajlo/datalore

## Overview

##### Project Type and Purpose
- Datalore is a Python‐based command‐line data‐analysis tool that leverages Anthropic’s Claude API for NL processing and integrates common data‐analysis libraries (pandas, numpy, matplotlib, etc.).
- It provides natural‐language interaction, data loading/cleaning/EDA/statistics/visualization, and a sandboxed custom‐code executor.

##### File Structure
- Top‐level scripts:
- datalore.py – main entrypoint (CLI orchestrator)
- redatalore.py – a variant or refactoring of the main script
- rnd.py – helper for randomness or utilities
- Configuration:
- requirements.txt – lists dependencies (pandas, anthropic, etc.)
- .env (not in tree, but mentioned in README) – stores the API key
- Persistence/Cache:
- .cursor/rules/data-rules.mdc – perhaps conversation history or rules storage
- No separate frontend or backend directories—this is a single‐process CLI application.

##### README for Architecture Clues
- Key components:
- CLI interface
- Natural‐language processing via external Anthropic Claude API
- Data‐loading modules (CSV, Excel, JSON)
- Data‐preprocessing/cleaning
- EDA/statistics modules
- Visualization modules
- Sandboxed custom code execution
- Conversation history management
- Technical stack: Python 3.7+, Anthropic API, pandas, matplotlib, etc.

##### Main Components
a. User & CLI Layer
– Terminal / Console (User input & output)
b. Core Orchestrator
– datalore.py (or redatalore.py)
– Input parser / command dispatcher
– Conversation history manager (.cursor/rules)
c. External Service
– Anthropic Claude API (cloud)
d. Data Modules (local in‐process)
– Data Loader (CSV, Excel, JSON)
– Preprocessing & Cleaning
– EDA & Statistical Analysis
– Visualization
e. Custom Code Executor
– Sandboxed Python runtime (executes user‐supplied code safely)
f. Configuration & Dependencies
– .env (ANTHROPIC_API_KEY)
– requirements.txt (pandas, numpy, matplotlib, anthropic, etc.)
g. Data Storage
– In‐memory DataFrames
– Local filesystem (source data files)

##### Relationships & Data Flows
- User → CLI → Core Orchestrator
- Core Orchestrator ⇄ Anthropic Claude API (for NL understanding and suggestions)
- Core → Data Loader → In‐memory DataFrame
- Core → Preprocessing / EDA / Statistics → DataFrame
- Core → Visualization → Matplotlib/Seaborn → Console or file outputs
- Core → Code Executor → Sandbox → Modified DataFrame
- Core ↔ Conversation History (append user queries & model responses)

##### Architectural Patterns & Principles
- Single‐process modular design (no microservices)
- Clear separation of concerns: CLI, NLP, data modules, sandbox execution
- Dependency injection for the API client via configuration (.env)
- Safety layer around code execution
- Pluggable data‐loading interfaces

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Actor
    User["User / Terminal"]:::external

    %% CLI & Orchestrator
    subgraph "CLI & Orchestrator"
        CLI["datalore.py"]:::internal
        ALTCLI["redatalore.py"]:::internal
        Parser["Input Parser / Dispatcher"]:::internal
    end

    %% Conversation History
    subgraph "Conversation History"
        HistoryDir[".cursor/rules/"]:::storage
        HistoryFile["data-rules.mdc"]:::storage
    end

    %% Data Modules
    subgraph "Data Modules"
        Loader["Data Loader (CSV/Excel/JSON)"]:::internal
        Preproc["Preprocessing & Cleaning"]:::internal
        EDA["EDA & Statistics"]:::internal
        Viz["Visualization (Matplotlib/Seaborn)"]:::internal
    end

    %% Code Execution
    Sandbox["Sandboxed Code Executor"]:::internal

    %% External Service
    Anthropic["Anthropic Claude API"]:::external

    %% Data Storage
    subgraph "Data Storage"
        MemDF["In-memory DataFrame"]:::storage
        FS["Filesystem (Source Data Files)"]:::storage
    end

    %% Configuration
    subgraph "Configuration"
        Env[".env (API Key)"]:::config
        Req["requirements.txt"]:::config
        Doc["README.md"]:::config
    end

    %% Connections
    User --> CLI
    CLI --> Parser
    Parser -->|"NL query"| Anthropic
    Anthropic -->|"NL response"| Parser
    Parser --> Loader
    Loader --> MemDF
    Parser --> Preproc
    Preproc --> MemDF
    Parser --> EDA
    EDA --> MemDF
    Parser --> Viz
    Viz -->|"Plot image"| FS
    Parser --> Sandbox
    Sandbox --> MemDF
    Parser --> HistoryDir
    HistoryDir --> HistoryFile
    Parser <-->|"append/log"| HistoryFile
    Env --> CLI
    Req --> CLI
    Doc --> CLI

    %% Click Events
    click CLI "https://github.com/micic-mihajlo/datalore/blob/main/datalore.py"
    click ALTCLI "https://github.com/micic-mihajlo/datalore/blob/main/redatalore.py"
    click Sandbox "https://github.com/micic-mihajlo/datalore/blob/main/rnd.py"
    click HistoryDir "https://github.com/micic-mihajlo/datalore/tree/main/.cursor/rules/"
    click HistoryFile "https://github.com/micic-mihajlo/datalore/blob/main/.cursor/rules/data-rules.mdc"
    click Req "https://github.com/micic-mihajlo/datalore/blob/main/requirements.txt"
    click Doc "https://github.com/micic-mihajlo/datalore/blob/main/README.md"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef external fill:#98FB98,stroke:#333,stroke-width:1px
    classDef storage fill:#FFD700,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
```
