How vitalops/datatune works
Project type and purpose
• Datatune is an open-source Python library (not a full web app) for performing data transformations (map, filter) on large datasets via natural-language prompts and LLM calls.
• Its core use case: pipeline Dask DataFrames through operations that invoke an LLM (OpenAI, Ollama, Azure).
File structure
• Top-level package datatune
• core/ – implements the Map, Filter classes and the finalize utility
• llm/ – defines the LLM abstraction and provider implementations
• examples/ – demonstration notebooks
• tests/ – unit tests for each operation and dataset integration
• CI/config via pyproject.toml, .gitignore, LICENSE, README.md
README for additional insights
• Describes the API: instantiate an LLM client, call Map(...) or Filter(...) with a prompt, pass in a Dask DataFrame, then call finalize().
• Mentions multiple providers (OpenAI, Azure, Ollama) and data‐compatibility (Dask ↔ pandas).
a. Main components:
• Client code (user’s Python script or notebook)
• Datatune Core (Map, Filter, finalize)
• LLM Module (LLM abstraction + concrete providers)
• External LLM Services (OpenAI API, Ollama daemon, Azure endpoints)
• Data Sources/Sinks (CSV files, Dask DataFrames, output CSV)
b. Relationships & interactions:
1. Client code loads DataFrame (Dask) from CSV
2. Client constructs an LLM instance (e.g. LLM(model_name=…))
3. /Filter operators in Datatune Core accept (LLM, DataFrame)
4. Datatune Core partitions the DataFrame, serializes each row/partition, sends prompt + data to LLM Module
5. LLM Module routes the request to the chosen External Service
6. Responses flow back to Datatune Core, which stitches them into new columns or filters rows
7. Client calls finalize(), producing a cleaned Dask DataFrame → writes to CSV
c. Architectural patterns & principles:
• Pipeline pattern: each operation (Map, Filter) is a composable function in a data pipeline
• Dependency inversion: Core logic depends on an LLM interface, not concrete APIs
• Scalability via Dask: parallel, chunked data processing
d. Key technologies:
• Python
• Dask DataFrames
• HTTP/REST to LLM services (OpenAI, Azure) or local gRPC for Ollama
• pytest for tests