# dataloop-ai-apps/active-learning architecture

> Architecture diagram of the GitHub repository dataloop-ai-apps/active-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataloop-ai-apps/active-learning
- Repository: https://github.com/dataloop-ai-apps/active-learning

## Overview

##### Project type and purpose
- This is an open-source “app” (plugin) for the Dataloop platform that provides three custom Active Learning nodes (data split, create-new-model, compare-models).
- It bundles Python modules (node implementations), JSON pipeline-config templates, and Vue.js panels (UI) to integrate into Dataloop’s pipeline builder and runtime.

##### File structure
- Top-level files:
– Dockerfile, .dataloop.cfg, requirements.txt: build and package the app as a container for Dataloop runtime.
– modules/ .py files: business logic for each node.
– pipeline_configs/: JSON templates for node parameters and filters.
– panels/dataSplit/: static build of the Vue.js panel for the DataSplit node.
– srcs/dataSplit/: source code (Vue, TypeScript, Vite) for that panel.
– docs/, assets/: documentation and images illustrating workflows.

##### Additional insights from README
- Each node’s inputs, outputs, parameters, and underlying behavior are documented.

– User & Dataloop Web UI (green)
– Custom Panels (Vue.js SPA) for node configuration (blue)
– Pipeline Orchestrator (Dataloop engine) (orange)
– Node Runtimes (Python modules in Docker) (purple)
– Dataloop Backend Services: Model Registry, Dataset Storage, Metadata Service (gray)
– Pipeline Config Templates (JSON)

1. User interacts with Dataloop Web UI → loads Custom Panel UI from panels/dataSplit (and others).
2. Panel UI reads/writes JSON templates from pipeline_configs to populate form fields.
3. User submits node configuration → Dataloop Orchestrator stores config.
4. At runtime, Orchestrator spins up Docker container (built from Dockerfile) hosting Python modules.
5. Node code calls Dataloop SDK (dl.Model, dl.Dataset) to perform:
– data_split: tag items in storage
– create_new_model: clone & fine-tune model
– model_compare: fetch metrics, compute AUC-PR, output decision
6. Results (tags, new model entity, winning_model) flow back to Pipeline Orchestrator → shown in UI.

c. Architectural patterns & principles
– Plugin architecture: UI pan­els and Python modules are loosely coupled via the Dataloop platform.
– Separation of concerns:
- Frontend (Vue.js panels) only for config.
- Backend (Python modules) only for execution logic.
– Containerization: each node runs in an isolated Docker environment.
– Extensibility: new nodes can be added by dropping modules, panels, configs.

d. Key technologies & frameworks
– Frontend: Vue.js, TypeScript, Vite
– Backend: Python, Dataloop SDK (dl.*)
– Configuration: JSON templates in pipeline_configs
– Packaging: Docker, .dataloop.cfg
– CI/CD & versioning: GitHub repo, bumpversion, CONTRIBUTING guidelines

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph: User & UI
    subgraph "User & UI"
        direction TB
        User["User"]:::green
        Dataloop_Web_UI["Dataloop Web UI"]:::green
    end

    %% Subgraph: Custom Panel Layer
    subgraph "Custom Panel Layer"
        direction TB
        DataSplit_Panel["DataSplit Panel (Vue.js)"]:::blue
        DataSplit_Panel_Assets["Panel Assets"]:::blue
        DataSplit_Panel_Source["DataSplit Panel Source"]:::blue
    end

    %% Subgraph: Pipeline Config Templates
    subgraph "Pipeline Config Templates"
        direction TB
        train_subset["train_subset_filter.json"]:::config
        validation_subset["validation_subset_filter.json"]:::config
        test_subset["test_subset_filter.json"]:::config
        compare_configs["compare_configurations.json"]:::config
    end

    %% Subgraph: Control Plane
    subgraph "Pipeline Orchestrator"
        direction TB
        Orchestrator["Dataloop Orchestrator"]:::orange
    end

    %% Subgraph: Execution Layer
    subgraph "Node Runtime Containers"
        direction TB
        Docker_Container["Docker Container (Node Runtime)"]:::purple
        DataSplit_Module["DataSplit Module (Python)"]:::purple
        CreateNewModel_Module["CreateNewModel Module (Python)"]:::purple
        ModelCompare_Module["ModelCompare Module (Python)"]:::purple
    end

    %% Subgraph: Platform Services
    subgraph "Dataloop Backend Services"
        direction TB
        ModelRegistry["Model Registry"]:::gray
        DatasetStorage["Dataset Storage"]:::gray
        MetadataService["Metadata Service"]:::gray
    end

    %% Subgraph: Documentation & Descriptor
    subgraph "Docs & Descriptor"
        direction TB
        DataSplit_Doc["DataSplit README"]:::config
        ModelCompare_Doc["ModelCompare README"]:::config
        App_Descriptor["dataloop.json"]:::config
    end

    %% Relationships
    User --> Dataloop_Web_UI
    Dataloop_Web_UI --> DataSplit_Panel
    DataSplit_Panel --> DataSplit_Panel_Assets
    DataSplit_Panel --> DataSplit_Panel_Source
    DataSplit_Panel_Source --> train_subset
    DataSplit_Panel_Source --> validation_subset
    DataSplit_Panel_Source --> test_subset
    DataSplit_Panel_Source --> compare_configs

    DataSplit_Panel -->|"submit config"| Orchestrator
    train_subset --> Orchestrator
    validation_subset --> Orchestrator
    test_subset --> Orchestrator
    compare_configs --> Orchestrator

    Orchestrator -->|"launch"| Docker_Container
    Docker_Container --> DataSplit_Module
    Docker_Container --> CreateNewModel_Module
    Docker_Container --> ModelCompare_Module

    DataSplit_Module -->|dl.Dataset.filter/tag| DatasetStorage
    DataSplit_Module -->|tag items| MetadataService

    CreateNewModel_Module -->|dl.Model.clone & fine-tune| ModelRegistry

    ModelCompare_Module -->|fetch metrics| ModelRegistry
    ModelCompare_Module -->|compute AUC-PR| MetadataService

    DataSplit_Module -->|"results"| Orchestrator
    CreateNewModel_Module -->|"results"| Orchestrator
    ModelCompare_Module -->|"results"| Orchestrator

    Orchestrator -->|"display results"| Dataloop_Web_UI

    DataSplit_Doc
    ModelCompare_Doc
    App_Descriptor

    %% Click Events
    click DataSplit_Module "https://github.com/dataloop-ai-apps/active-learning/blob/main/modules/data_split.py"
    click CreateNewModel_Module "https://github.com/dataloop-ai-apps/active-learning/blob/main/modules/create_new_model.py"
    click ModelCompare_Module "https://github.com/dataloop-ai-apps/active-learning/blob/main/modules/model_compare.py"
    click DataSplit_Panel "https://github.com/dataloop-ai-apps/active-learning/blob/main/panels/dataSplit/index.html"
    click DataSplit_Panel_Assets "https://github.com/dataloop-ai-apps/active-learning/tree/main/panels/dataSplit/assets/"
    click DataSplit_Panel_Source "https://github.com/dataloop-ai-apps/active-learning/tree/main/srcs/dataSplit/"
    click train_subset "https://github.com/dataloop-ai-apps/active-learning/blob/main/pipeline_configs/train_subset_filter.json"
    click validation_subset "https://github.com/dataloop-ai-apps/active-learning/blob/main/pipeline_configs/validation_subset_filter.json"
    click test_subset "https://github.com/dataloop-ai-apps/active-learning/blob/main/pipeline_configs/test_subset_filter.json"
    click compare_configs "https://github.com/dataloop-ai-apps/active-learning/blob/main/pipeline_configs/compare_configurations.json"
    click Docker_Container "https://github.com/dataloop-ai-apps/active-learning/tree/main/Dockerfile"
    click DataSplit_Doc "https://github.com/dataloop-ai-apps/active-learning/blob/main/docs/data_split/README.md"
    click ModelCompare_Doc "https://github.com/dataloop-ai-apps/active-learning/blob/main/docs/model_compare/README.md"
    click App_Descriptor "https://github.com/dataloop-ai-apps/active-learning/blob/main/dataloop.json"

    %% Styles
    classDef green fill:#ddffdd,stroke:#006600,color:#006600
    classDef blue fill:#ddddff,stroke:#000066,color:#000066
    classDef orange fill:#ffedcc,stroke:#cc6600,color:#cc6600
    classDef purple fill:#f0ddff,stroke:#660099,color:#660099
    classDef gray fill:#eeeeee,stroke:#666666,color:#666666
    classDef config fill:#f8f8f8,stroke:#bbbbbb,color:#333333
```
