# clearml/clearml architecture

> Architecture diagram of the GitHub repository clearml/clearml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/clearml/clearml
- Repository: https://github.com/clearml/clearml

## Overview

##### Project Type and Purpose
- This is an MLOps/Experiment Management suite for AI/ML workflows. It supports experiment tracking, remote execution (via agents), data and model management, and advanced automation (e.g., hyper-parameter optimization and orchestration).
- The README outlines five key modules (experiment management, MLOps/LLMOps, data management, model serving, and reports) along with a clear explanation of the roles of the server and agent components.

##### Analyzing the File Structure for Architectural Clues
- Top-level directories include:
- clearml/automation – Contains modules for automated scheduling, optimization (Optuna and hpbandster), and cloud drivers (AWS, cloud, auto_scaler). This indicates a focus on orchestration and remote resource management.
- clearml/backend_api – Contains API proxy, configuration files, schemas, service implementations (multiple service versions such as v2_9, v2_13, v2_20, v2_23). This is the heart of the server-side API.
- clearml/backend_config – Handles configuration management for the backend.
- clearml/backend_interface – Provides abstract interfaces for logging, metrics, session handling, and tasks.
- clearml/binding – Offers integrations with various frameworks (e.g., PyTorch, TensorFlow, LightGBM) and command line utilities and bindings (click, argparse, fire, etc.). This is the client–side integration component.
- clearml/cli – Contains CLI tools for configuration, data handling, debugging, hyper-parameter optimization, and task management.
- Other directories like clearml/datasets, clearml/debugging, clearml/storage, and clearml/utilities suggest additional support modules (data versioning, logging, caching, and utility helpers).
- The presence of configuration files (e.g., .flake8, setup.py, setup.cfg, dev-requirements.txt) and GitHub workflow files indicates strong support for deployment and CI/CD practices.

##### Key Components
a. Client / SDK
- Represented by the Python package (clearml package, clearml/binding). This is the component that a user integrates into his/her ML code. It is responsible for logging experiments, capturing artifacts, and reporting metrics.
- Integrations with various frameworks (PyTorch, TensorFlow, Keras, etc.), which can be depicted as external “framework” modules that connect to the client.
b. ClearML Server (Backend)
- Shows the central repository for experiments, models, logs, and configuration.
- Within the server, illustrate the backend API layer. You may break this into subcomponents:
- API Proxy (clearml/backend_api/api_proxy.py)
- Config and Schema modules (clearml/backend_api/config and clearml/backend_api/schema)
- Versioned service endpoints (v2_9, v2_13, v2_20, v2_23), indicating how the system is backward-compatible and maintains several API versions.
- Session management components (clearml/backend_api/session) for handling API calls and authentication.
c. ClearML Agent
- This component orchestrates the remote execution of tasks and runs experiments on remote machines. Files under clearml/automation (such as controller.py, scheduler.py, trigger.py, and auto_scaler.py) and examples in the clearml_agent folder point to this functionality.
- It should be shown as a separate entity that interacts with the server (e.g., “pulls tasks”) and may interface with cloud resources (AWS) via modules like aws_driver.py.
d. Additional Modules/Subsystems
- Utilities and Logging: Modules under clearml/debugging and clearml/utilities dealing with logging, GPU monitoring, file management, and distributed operations.
- CLI Tools: Represent a command line interface component (clearml/cli) for managing tasks, configurations, and remote monitoring.

##### Relationships and Interactions
- Communication from the Agent to the Server: the Agent queries the server for pending tasks (using scheduling components) and reports back completion details.
- For orchestration and automation, show internal communications within the Backend (API, Config, Session, etc.) and from the automation modules (e.g., job orchestration, optimization algorithms) to the agent.

##### Architectural Patterns and Design Principles
- Modular Design: Each folder (automation, backend_api, binding, cli) is a module with a distinct responsibility. Represent each as separate blocks.
- Service-Oriented: The backend services follow a design that is almost microservice-like, with clear separation between logging, authentication, events, and worker management.
- Integration Patterns: The various bindings and CLI tools suggest a high degree of flexibility for integrating with many frameworks and external computational resources.

- ClearML Python Package (Client/SDK) that users integrate into their experimental scripts.
- ClearML Server and break it down to show the API, configuration, and session management layers.
- ClearML Agent’s role in task execution and orchestration.
- Additional tooling modules such as the CLI, automation modules (scheduler, trigger, auto_scaler), and integrations (bindings, external frameworks).
- Data flow (from client to server for logging experiments, from server to agent for task dispatching).
- Client-side components
- Server-side components
- Orchestration/Automation components
- Descriptive notes next to components if needed to detail their responsibilities.

## Diagram (Mermaid)

```mermaid
flowchart TD

  %% Client/SDK Layer
  subgraph "Client/SDK Layer"
      A1["Python SDK/Client"]:::client
      A2["Bindings/Integrations"]:::client
      A1 -->|"uses"| A2
  end

  %% ClearML Server (Backend API)
  subgraph "ClearML Server (Backend API)"
      B1["API Proxy"]:::server
      B2A["Config"]:::server
      B2B["Schema"]:::server
      B3["Versioned Services (v2_9,v2_13,v2_20,v2_23)"]:::server
      B4["Session Management"]:::server
      B1 -->|"loads"| B2A
      B1 -->|"loads"| B2B
      B2A -->|"routes_to"| B3
      B2B -->|"routes_to"| B3
      B3 -->|"manages"| B4
  end

  %% Agent/Automation Layer
  subgraph "Agent/Automation Layer"
      C1["ClearML Agent (Controller, Scheduler, Trigger, Auto-Scaler, AWS Driver)"]:::agent
  end

  %% Data/Artifact Management
  subgraph "Data/Artifact Management"
      D1["Datasets"]:::data
      D2["Storage"]:::data
      D1 -->|"integrates_with"| D2
  end

  %% CLI Tools
  subgraph "CLI Tools"
      E1["ClearML CLI Tools"]:::cli
  end

  %% Interactions
  A2 -->|"log_experiments"| B1
  B1 -->|"dispatch_tasks"| C1
  C1 -->|"report_results"| B1
  A2 ---|"submit_datasets"| D1
  B3 ---|"store_artifacts"| D2
  E1 -->|"administers"| B1

  %% Click Events
  click A1 "https://github.com/clearml/clearml/blob/master/clearml/__init__.py"
  click A2 "https://github.com/clearml/clearml/tree/master/clearml/binding"
  click B1 "https://github.com/clearml/clearml/blob/master/clearml/backend_api/api_proxy.py"
  click B2A "https://github.com/clearml/clearml/tree/master/clearml/backend_api/config/"
  click B2B "https://github.com/clearml/clearml/tree/master/clearml/backend_api/schema/"
  click B3 "https://github.com/clearml/clearml/tree/master/clearml/backend_api/services/"
  click B4 "https://github.com/clearml/clearml/tree/master/clearml/backend_api/session/"
  click C1 "https://github.com/clearml/clearml/tree/master/clearml/automation"
  click D1 "https://github.com/clearml/clearml/tree/master/clearml/datasets"
  click D2 "https://github.com/clearml/clearml/tree/master/clearml/storage"
  click E1 "https://github.com/clearml/clearml/tree/master/clearml/cli"

  %% Styles
  classDef client fill:#f0f8ff,stroke:#000,stroke-width:2px;
  classDef server fill:#fffacd,stroke:#000,stroke-width:2px;
  classDef agent fill:#dcedc1,stroke:#000,stroke-width:2px;
  classDef data fill:#ffcdd2,stroke:#000,stroke-width:2px;
  classDef cli fill:#e1bee7,stroke:#000,stroke-width:2px;
```
