How future-house/data-analysis-crow works
Project Type and Purpose
• This is a Python-based AI agent framework (“Data Analysis Crow”) that orchestrates iterative Jupyter-notebook–driven data analyses.
• It provides:
• A CLI/API to accept datasets + prompts
• A notebook-building engine using Jinja2 templates
• Integration with LLMs (via litellm)
• A Dockerized environment for bioinformatics packages
• Scripts to run BixBench benchmark experiments
File Structure
• Top-level:
• src/fhda – core library (config, dataset handling, model logic, storage, templates, utils)
• src/expts – experiment harness (client/server, dsbench, eval, training)
• src/scripts/bixbench_evaluation – evaluation runner (YAML configs, run.sh)
• tutorial – example notebooks and datasets
• CI/config files (.github, pre-commit, Dockerfile)
• Key patterns:
• Modular library (src/fhda)
• Experiment orchestration (src/expts + scripts)
• Notebook templating (Jinja2 in src/fhda/templates)
• Config via YAML (.env, dev.yaml, server.yaml, runner.yaml)
Extract Architecture Insights from README
• Core flow:
1. User invokes CLI / Python API with dataset + prompt
2. Agent loads config & dataset definitions (dataset.py)
3. Agent queries LLM APIs (litellm clients)
4. Agent uses templates to build notebook cells (Jinja2)
5. Agent executes notebook (Jupyter server or papermill inside Docker)
6. Results & intermediate notebooks are stored/logged
7. For BixBench: runner.sh drives experiments, collects trajectories
Main Components
a. User Interface Layer
– CLI / Python API entry point
– Configuration subsystem (.env, dev.yaml, runner.yaml, server.yaml)
b. Agent Core
– Experiment harness (src/expts: client.py, server.py, dsbench.py, training.py, eval.py)
– fhda library modules (config.py, dataset.py, models.py, storage.py, utils.py)
– Prompt & template renderer (src/fhda/prompts.py + src/fhda/templates)
c. Execution Environment
– Notebook engine (Jupyter server or papermill process)
– Docker container (Dockerfile.pinned) with bioinformatics packages
d. External Services & APIs
– LLM providers (OpenAI, Anthropic, etc. via litellm)
– Optional storage (local filesystem or remote storage via storage.py)
– BixBench benchmark repo (scripts/bixbench_evaluation)
e. Data Stores
– Input datasets (local files, Hugging Face)
– Generated notebooks & outputs (tutorial/tmp_results_dir or user-specified output dir)
– Logs/metrics (expt_logging.py)
Relationships and Data Flows
• User → CLI/API → loads Config
• Agent Core → Dataset module → fetches dataset
• Agent Core → Prompt module → sends prompt to LLM API
• LLM API → returns code snippets / analysis steps
• Agent Core → Template renderer → constructs notebook cells
• Agent Core → Notebook Engine → executes notebook
• Notebook Engine → writes results/notebooks → Storage
• Experiment Runner (scripts) → orchestrates repeated agent runs → aggregates metrics
Architectural Patterns & Principles
• Layered modular design (separate config, data, model, template, execution)
• Plug-in style for different runtimes (Python/R/Bash) via notebook engine
• Experiment/Reproducibility focus (YAML configs, logging, scripts)
• Containerization for consistency (Dockerfile.pinned)