# dataloop-ai-apps/dtlpy-metrics architecture

> Architecture diagram of the GitHub repository dataloop-ai-apps/dtlpy-metrics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataloop-ai-apps/dtlpy-metrics
- Repository: https://github.com/dataloop-ai-apps/dtlpy-metrics

## Overview

##### Project Type and Purpose
- This is an open-source Python library/app (“dtlpymetrics”) that plugs into the Dataloop annotation platform.
- Its purpose is to calculate and upload annotation quality scores (raw, overall, confusion, item) for image/video annotation tasks.
- It provides standalone CLI entrypoint (runner.py) and custom pipeline nodes for real-time scoring within Dataloop pipelines.

a. CLI Runner
- runner.py – invokes dtlpymetrics as a script.
- build.sh, Dockerfile – packaging layers for containerized execution.
b. Core Package (dtlpymetrics)
- dtlpy_scores.py – orchestrates Dataloop SDK calls and scoring flows.
- evaluating/ (models.py, tasks.py) – “reference vs. assignee” comparisons and confusion logic.
- scoring/ (models.py, tasks.py) – raw annotation scoring (IOU, label, attribute).
- utils/ – helper functions (geometry, matching, plotting).
- __version__.py, __init__.py – package metadata.
c. External Dependencies
- Dataloop Platform (API server + item/annotation storage)
- Python libs: dtlpy SDK, numpy, shapely, OpenCV, requests, etc.
d. Configuration & CI
- dataloop.json – Dataloop project credentials/config.
- requirements.txt, setup.py – dependency declaration and packaging.
- GitHub workflows (.github/workflows) – CI tests, build, publish, tag.
e. Tests & Assets
- tests/ – unit and integration tests using stored JSON annotation assets.
- docs/ – design and function documentation (metrics.ipynb, dtlpymetrics_fxns.md).

##### Relationships and Data Flows
- CLI Runner → imports dtlpymetrics → loads dataloop.json → authenticates to Dataloop API.
- dtlpy_scores.py → calls scoring.tasks (raw scores) and evaluating.tasks (confusion, overall).
- scoring.tasks → use utils (matching, geometry) → calculate annotation-level scores.
- evaluating.tasks → aggregate raw scores → compute user_confusion, label_confusion, item_overall.
- dtlpy_scores uploads computed scores back to Dataloop via dtlpy SDK.
- In-pipeline Custom Nodes (tasks.py) are invoked by Dataloop when a task’s annotations complete, mirroring CLI flow but event-driven by the pipeline engine.

##### Architectural Patterns & Design Principles
- Modular separation: “scoring” vs. “evaluating” vs. “utils” layers.
- Pluggable pipeline design: CLI and SDK-driven nodes share core logic.
- Single responsibility: each task.py handles one type of score.
- Config-driven: dataloop.json and environment variables drive authentication and pipeline behavior.
- Testable: assets under tests/ simulate real annotation payloads; enables CI pipelines to verify scoring outputs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers and Components
    subgraph "Local Code" 
        direction TB
        CLI["CLI Runner"]:::code
        runner["runner.py"]:::code
        dockerfile["Dockerfile"]:::code
        buildsh["build.sh"]:::code
        subgraph "dtlpymetrics Package"
            direction TB
            init["__init__.py"]:::code
            version["__version__.py"]:::code
            scores["dtlpy_scores.py"]:::code
            subgraph "scoring"
                direction TB
                sc_models["models.py"]:::code
                sc_tasks["tasks.py"]:::code
            end
            subgraph "evaluating"
                direction TB
                ev_models["models.py"]:::code
                ev_tasks["tasks.py"]:::code
            end
            subgraph "utils"
                direction TB
                util_init["__init__.py"]:::code
                dl_helpers["dl_helpers.py"]:::code
                matching["matching.py"]:::code
                plotting["plotting.py"]:::code
            end
        end
    end

    subgraph "Configuration & CI" 
        direction TB
        cfg["dataloop.json"]:::config
        reqs["requirements.txt"]:::config
        setup["setup.py"]:::config
        workflows[".github/workflows"]:::config
    end

    subgraph "Documentation & Tests"
        direction TB
        docsmd["dtlpymetrics_fxns.md"]:::docs
        metricsnb["metrics.ipynb"]:::docs
        test_py["test_main.py"]:::tests
        assets["assets/"]:::tests
    end

    subgraph "External Systems"
        direction TB
        SDK["Dataloop SDK"]:::external
        API["Dataloop API"]:::external
        Platform["Dataloop Platform"]:::external
    end

    subgraph "Pipeline Integration"
        direction TB
        Event["Pipeline Event"]:::code
        Node["Custom Pipeline Node"]:::code
    end

    %% Data Flows
    CLI -->|imports| scores
    runner -->|executes| CLI
    CLI -->|loads config| cfg
    scores -->|calls raw scoring| sc_tasks
    sc_tasks -->|uses| dl_helpers
    sc_tasks -->|uses| matching
    sc_tasks -->|uses| plotting
    sc_tasks -->|computes IOU/label| sc_models
    scores -->|calls evaluating| ev_tasks
    ev_tasks -->|aggregates| ev_models
    scores -->|uploads scores| SDK
    SDK -->|REST API| API
    API -->|stores annotations| Platform

    Event -->|trigger| Node
    Node -->|invokes| scores
    Node -->|uploads| SDK

    %% Supporting Components
    scores --> cfg
    workflows -->|CI runs tests| test_py
    test_py -->|uses assets| assets
    docsmd -->|documents| sc_tasks
    metricsnb -->|documents| ev_tasks

    %% Click Events
    click runner "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/runner.py"
    click dockerfile "https://github.com/dataloop-ai-apps/dtlpy-metrics/tree/main/Dockerfile"
    click buildsh "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/build.sh"
    click init "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/__init__.py"
    click version "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/__version__.py"
    click scores "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/dtlpy_scores.py"
    click sc_models "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/scoring/models.py"
    click sc_tasks "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/scoring/tasks.py"
    click ev_models "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/evaluating/models.py"
    click ev_tasks "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/evaluating/tasks.py"
    click util_init "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/utils/__init__.py"
    click dl_helpers "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/utils/dl_helpers.py"
    click matching "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/utils/matching.py"
    click plotting "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dtlpymetrics/utils/plotting.py"
    click cfg "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/dataloop.json"
    click reqs "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/requirements.txt"
    click setup "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/setup.py"
    click workflows "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/.github/workflows/ci-testing.yml"
    click workflows "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/.github/workflows/create_tag.yml"
    click workflows "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/.github/workflows/publish.yml"
    click docsmd "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/docs/dtlpymetrics_fxns.md"
    click metricsnb "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/docs/metrics.ipynb"
    click test_py "https://github.com/dataloop-ai-apps/dtlpy-metrics/blob/main/tests/test_main.py"
    click assets "https://github.com/dataloop-ai-apps/dtlpy-metrics/tree/main/tests/assets/"

    %% Styles
    classDef code fill:#D0E6FF,stroke:#004085,color:#004085;
    classDef config fill:#FFF3CD,stroke:#856404,color:#856404;
    classDef external fill:#D4EDDA,stroke:#155724,color:#155724;
    classDef docs fill:#E2E3E5,stroke:#6C757D,color:#6C757D;
    classDef tests fill:#F8D7DA,stroke:#721C24,color:#721C24;
```
