# vitalops/datatune architecture

> Architecture diagram of the GitHub repository vitalops/datatune, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vitalops/datatune
- Repository: https://github.com/vitalops/datatune

## Overview

##### Project type and purpose
- Datatune is an open-source Python library (not a full web app) for performing data transformations (map, filter) on large datasets via natural-language prompts and LLM calls.
- Its core use case: pipeline Dask DataFrames through operations that invoke an LLM (OpenAI, Ollama, Azure).

##### File structure
- Top-level package `datatune`
- `core/` – implements the Map, Filter classes and the finalize utility
- `llm/` – defines the LLM abstraction and provider implementations
- `examples/` – demonstration notebooks
- `tests/` – unit tests for each operation and dataset integration
- CI/config via `pyproject.toml`, `.gitignore`, `LICENSE`, `README.md`

##### README for additional insights
- Describes the API: instantiate an LLM client, call `Map(...)` or `Filter(...)` with a prompt, pass in a Dask DataFrame, then call `finalize()`.
- Mentions multiple providers (OpenAI, Azure, Ollama) and data‐compatibility (Dask ↔ pandas).

a. Main components:
- Client code (user’s Python script or notebook)
- Datatune Core (Map, Filter, finalize)
- LLM Module (LLM abstraction + concrete providers)
- External LLM Services (OpenAI API, Ollama daemon, Azure endpoints)
- Data Sources/Sinks (CSV files, Dask DataFrames, output CSV)
b. Relationships & interactions:
1. Client code loads DataFrame (Dask) from CSV
2. Client constructs an LLM instance (e.g. `LLM(model_name=…)`)
3. /Filter operators in Datatune Core accept (LLM, DataFrame)
4. Datatune Core partitions the DataFrame, serializes each row/partition, sends prompt + data to LLM Module
5. LLM Module routes the request to the chosen External Service
6. Responses flow back to Datatune Core, which stitches them into new columns or filters rows
7. Client calls `finalize()`, producing a cleaned Dask DataFrame → writes to CSV
c. Architectural patterns & principles:
- Pipeline pattern: each operation (`Map`, `Filter`) is a composable function in a data pipeline
- Dependency inversion: Core logic depends on an LLM interface, not concrete APIs
- Scalability via Dask: parallel, chunked data processing
d. Key technologies:
- Python
- Dask DataFrames
- HTTP/REST to LLM services (OpenAI, Azure) or local gRPC for Ollama
- pytest for tests

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client"
        CC["Client Code (Notebook)"]:::client
    end

    %% Data Sources
    CSV_SRC[(CSV Source)]:::csv
    CSV_SRC -->|"dask.read_csv"| DF["Dask DataFrame"]:::data

    %% Datatune Core Components
    subgraph "Datatune Core"
        direction TB
        C["Constants"]:::core
        OP["Operator Base & finalize"]:::core
        M["Map"]:::core
        F["Filter"]:::core
    end

    %% LLM Module
    LLM["LLM Abstraction & Providers"]:::core

    %% External LLM Services
    subgraph "External LLM Services"
        direction TB
        OAI["OpenAI API"]:::external
        AZ["Azure Endpoint"]:::external
        OLL["Ollama"]:::external
    end

    %% Data Sink
    DF --> M
    DF --> F
    M -->|"prompt + data"| LLM
    F -->|"prompt + data"| LLM
    LLM -->|"HTTP"| OAI
    LLM -->|"HTTP"| AZ
    LLM -->|"gRPC"| OLL
    OAI -->|response| LLM
    AZ -->|response| LLM
    OLL -->|response| LLM
    LLM -->|LLM response| M
    LLM -->|LLM response| F
    M -->|"adds column / filters rows"| OP
    F -->|"adds column / filters rows"| OP
    CC -->|"call finalize"| OP
    OP -->|"finalize: clean & output"| DF_OUT["Cleaned Dask DataFrame"]:::data
    DF_OUT -->|"to_csv"| CSV_SINK[(CSV Sink)]:::csv

    %% Click Events
    click M "https://github.com/vitalops/datatune/blob/main/datatune/core/map.py"
    click F "https://github.com/vitalops/datatune/blob/main/datatune/core/filter.py"
    click OP "https://github.com/vitalops/datatune/blob/main/datatune/core/op.py"
    click C "https://github.com/vitalops/datatune/blob/main/datatune/core/constants.py"
    click LLM "https://github.com/vitalops/datatune/blob/main/datatune/llm/llm.py"
    click CC "https://github.com/vitalops/datatune/blob/main/examples/ollama_datatune_example.ipynb"
    click tests "https://github.com/vitalops/datatune/tree/main/tests/"
    click test_data "https://github.com/vitalops/datatune/tree/main/tests/test_data/"
    click config "https://github.com/vitalops/datatune/blob/main/pyproject.toml"
    click readme "https://github.com/vitalops/datatune/blob/main/README.md"
    click license "https://github.com/vitalops/datatune/tree/main/LICENSE"
    click gitignore "https://github.com/vitalops/datatune/blob/main/.gitignore"

    %% Styling
    classDef client fill:#f9f,stroke:#333,stroke-width:1px
    classDef core fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef external fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    classDef data fill:#eeeeee,stroke:#757575,stroke-width:1px
    classDef csv fill:#fff9c4,stroke:#fbc02d,stroke-width:1px
```
