# cmu-db/ottertune architecture

> Architecture diagram of the GitHub repository cmu-db/ottertune, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cmu-db/ottertune
- Repository: https://github.com/cmu-db/ottertune

## Overview

OtterTune’s workflow is to collect DBMS configuration and performance observations, upload them to the service, normalize them into stored tuning results, and use historical workload data to recommend better knob settings. The graph preserves the documented cross-session learning capability and the principal analysis families referenced by the task modules. Relationships involving unsampled modules are limited to references in the SOURCE INDEX; optional client-to-server transport is documented only at the controller’s upload entry point.

Main entry points: [`server/website/website/tasks/async_tasks.py`](https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/async_tasks.py) (Recommendation tasks), [`client/controller/src/main/java/com/controller/Main.java`](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/Main.java) (Controller runner), [`server/website/website/tasks/periodic_tasks.py`](https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/periodic_tasks.py) (Workload processing), [`server/website/website/models.py`](https://github.com/cmu-db/ottertune/blob/master/server/website/website/models.py) (Tuning records)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["DBMS controller"]
  node_main["Controller runner<br/>[Main.java]"]
  node_config["Controller config"]
  node_collector["DB collectors<br/>[DBCollector.java]"]
  node_uploader["Result uploader"]
  node_json["JSON utilities<br/>[JSONUtil.java]"]
end

subgraph group_service["Tuning service"]
  node_web["Service views<br/>[views.py]"]
  node_models[("Tuning records<br/>[models.py]")]
  node_dbparser["DBMS parser dispatch<br/>[parser.py]"]
  node_baseparser["Base DBMS parser<br/>[parser.py]"]
  node_vendorparser["Vendor parsers"]
  node_async["Recommendation tasks<br/>[async_tasks.py]"]
  node_periodic["Workload processing<br/>[periodic_tasks.py]"]
  node_celery["Task scheduling<br/>[celery.py]"]
  node_settings["Service settings<br/>[common.py]"]
end

subgraph group_analysis["Learning and analysis"]
  node_preprocess["Data preprocessing<br/>[preprocessing.py]"]
  node_cluster["Workload clustering<br/>[cluster.py]"]
  node_gp["Gaussian processes<br/>[gp_tf.py]"]
  node_nn["Neural networks<br/>[nn_tf.py]"]
  node_ddpg["DDPG policy<br/>[ddpg.py]"]
  node_gpr["GPR models<br/>[gpr_models.py]"]
end

node_operator(("DB operator"))
node_dbms[("Target DBMS")]

node_operator -->|"runs"| node_main
node_main -->|"loads"| node_config
node_main -->|"selects"| node_collector
node_collector -.->|"collects from"| node_dbms
node_main -->|"uploads results"| node_uploader
node_main -->|"parses config"| node_json
node_uploader -.->|"submits observations"| node_web
node_web -->|"uses records"| node_models
node_web -->|"parses DBMS data"| node_dbparser
node_dbparser -->|"dispatches by DBMS"| node_vendorparser
node_vendorparser -->|"extends parser"| node_baseparser
node_periodic -->|"aggregates results"| node_models
node_periodic -->|"uses preprocessing"| node_preprocess
node_periodic -->|"uses clustering"| node_cluster
node_periodic -->|"uses task app"| node_celery
node_async -->|"reads tuning data"| node_models
node_async -->|"cleans observations"| node_preprocess
node_async -->|"uses model"| node_gp
node_async -->|"uses model"| node_nn
node_async -->|"uses policy"| node_ddpg
node_async -->|"uses models"| node_gpr
node_async -->|"uses task app"| node_celery
node_settings -.->|"configures service"| node_models

click node_main "https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/Main.java"
click node_config "https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/ControllerConfiguration.java"
click node_collector "https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/collectors/DBCollector.java"
click node_uploader "https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/ResultUploader.java"
click node_json "https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/util/JSONUtil.java"
click node_web "https://github.com/cmu-db/ottertune/blob/master/server/website/website/views.py"
click node_models "https://github.com/cmu-db/ottertune/blob/master/server/website/website/models.py"
click node_dbparser "https://github.com/cmu-db/ottertune/blob/master/server/website/website/db/parser.py"
click node_baseparser "https://github.com/cmu-db/ottertune/blob/master/server/website/website/db/base/parser.py"
click node_vendorparser "https://github.com/cmu-db/ottertune/tree/master/server/website/website/db"
click node_async "https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/async_tasks.py"
click node_periodic "https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/periodic_tasks.py"
click node_celery "https://github.com/cmu-db/ottertune/blob/master/server/website/website/celery.py"
click node_preprocess "https://github.com/cmu-db/ottertune/blob/master/server/analysis/preprocessing.py"
click node_cluster "https://github.com/cmu-db/ottertune/blob/master/server/analysis/cluster.py"
click node_gp "https://github.com/cmu-db/ottertune/blob/master/server/analysis/gp_tf.py"
click node_nn "https://github.com/cmu-db/ottertune/blob/master/server/analysis/nn_tf.py"
click node_ddpg "https://github.com/cmu-db/ottertune/blob/master/server/analysis/ddpg/ddpg.py"
click node_gpr "https://github.com/cmu-db/ottertune/blob/master/server/analysis/gpr/gpr_models.py"
click node_settings "https://github.com/cmu-db/ottertune/blob/master/server/website/website/settings/common.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config,node_collector,node_uploader,node_json toneBlue
class node_web,node_models,node_dbparser,node_baseparser,node_vendorparser,node_async,node_periodic,node_celery,node_settings,node_dbms toneAmber
class node_preprocess,node_cluster,node_gp,node_nn,node_ddpg,node_gpr toneMint
class node_operator toneIndigo
```

## Components

### DBMS controller

- [Controller runner](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/Main.java): `client/controller/src/main/java/com/controller/Main.java`
- [Controller config](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/ControllerConfiguration.java): `client/controller/src/main/java/com/controller/ControllerConfiguration.java`
- [DB collectors](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/collectors/DBCollector.java): `client/controller/src/main/java/com/controller/collectors/DBCollector.java`
- [Result uploader](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/ResultUploader.java): `client/controller/src/main/java/com/controller/ResultUploader.java`
- [JSON utilities](https://github.com/cmu-db/ottertune/blob/master/client/controller/src/main/java/com/controller/util/JSONUtil.java): `client/controller/src/main/java/com/controller/util/JSONUtil.java`

### Tuning service

- [Service views](https://github.com/cmu-db/ottertune/blob/master/server/website/website/views.py): `server/website/website/views.py`
- [Tuning records](https://github.com/cmu-db/ottertune/blob/master/server/website/website/models.py): `server/website/website/models.py`
- [DBMS parser dispatch](https://github.com/cmu-db/ottertune/blob/master/server/website/website/db/parser.py): `server/website/website/db/parser.py`
- [Base DBMS parser](https://github.com/cmu-db/ottertune/blob/master/server/website/website/db/base/parser.py): `server/website/website/db/base/parser.py`
- [Vendor parsers](https://github.com/cmu-db/ottertune/tree/master/server/website/website/db): `server/website/website/db`
- [Recommendation tasks](https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/async_tasks.py): `server/website/website/tasks/async_tasks.py`
- [Workload processing](https://github.com/cmu-db/ottertune/blob/master/server/website/website/tasks/periodic_tasks.py): `server/website/website/tasks/periodic_tasks.py`
- [Task scheduling](https://github.com/cmu-db/ottertune/blob/master/server/website/website/celery.py): `server/website/website/celery.py`
- [Service settings](https://github.com/cmu-db/ottertune/blob/master/server/website/website/settings/common.py): `server/website/website/settings/common.py`

### Learning and analysis

- [Data preprocessing](https://github.com/cmu-db/ottertune/blob/master/server/analysis/preprocessing.py): `server/analysis/preprocessing.py`
- [Workload clustering](https://github.com/cmu-db/ottertune/blob/master/server/analysis/cluster.py): `server/analysis/cluster.py`
- [Gaussian processes](https://github.com/cmu-db/ottertune/blob/master/server/analysis/gp_tf.py): `server/analysis/gp_tf.py`
- [Neural networks](https://github.com/cmu-db/ottertune/blob/master/server/analysis/nn_tf.py): `server/analysis/nn_tf.py`
- [DDPG policy](https://github.com/cmu-db/ottertune/blob/master/server/analysis/ddpg/ddpg.py): `server/analysis/ddpg/ddpg.py`
- [GPR models](https://github.com/cmu-db/ottertune/blob/master/server/analysis/gpr/gpr_models.py): `server/analysis/gpr/gpr_models.py`

### Other components

- **DB operator**
- **Target DBMS**

## Connections

- DB operator → Controller runner: runs (evidence: `client/controller/src/main/java/com/controller/Main.java`)
- Controller runner → Controller config: loads (evidence: `client/controller/src/main/java/com/controller/Main.java`)
- Controller runner → DB collectors: selects (evidence: `client/controller/src/main/java/com/controller/Main.java`)
- DB collectors → Target DBMS: collects from
- Controller runner → Result uploader: uploads results (evidence: `client/controller/src/main/java/com/controller/Main.java`)
- Controller runner → JSON utilities: parses config (evidence: `client/controller/src/main/java/com/controller/Main.java`)
- Result uploader → Service views: submits observations
- Service views → Tuning records: uses records (evidence: `server/website/website/views.py`)
- Service views → DBMS parser dispatch: parses DBMS data (evidence: `server/website/website/views.py`)
- DBMS parser dispatch → Vendor parsers: dispatches by DBMS (evidence: `server/website/website/db/parser.py`)
- Vendor parsers → Base DBMS parser: extends parser (evidence: `server/website/website/db/myrocks/parser.py`)
- Workload processing → Tuning records: aggregates results (evidence: `server/website/website/tasks/periodic_tasks.py`)
- Workload processing → Data preprocessing: uses preprocessing (evidence: `server/website/website/tasks/periodic_tasks.py`)
- Workload processing → Workload clustering: uses clustering (evidence: `server/website/website/tasks/periodic_tasks.py`)
- Workload processing → Task scheduling: uses task app (evidence: `server/website/website/tasks/periodic_tasks.py`)
- Recommendation tasks → Tuning records: reads tuning data (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → Data preprocessing: cleans observations (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → Gaussian processes: uses model (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → Neural networks: uses model (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → DDPG policy: uses policy (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → GPR models: uses models (evidence: `server/website/website/tasks/async_tasks.py`)
- Recommendation tasks → Task scheduling: uses task app (evidence: `server/website/website/tasks/async_tasks.py`)
- Service settings → Tuning records: configures service
