How dataloop-ai-apps/dtlpy-metrics works
Project Type and Purpose
• This is an open-source Python library/app (“dtlpymetrics”) that plugs into the Dataloop annotation platform.
• Its purpose is to calculate and upload annotation quality scores (raw, overall, confusion, item) for image/video annotation tasks.
• It provides standalone CLI entrypoint (runner.py) and custom pipeline nodes for real-time scoring within Dataloop pipelines.
a. CLI Runner
• runner.py – invokes dtlpymetrics as a script.
• build.sh, Dockerfile – packaging layers for containerized execution.
b. Core Package (dtlpymetrics)
• dtlpy_scores.py – orchestrates Dataloop SDK calls and scoring flows.
• evaluating/ (models.py, tasks.py) – “reference vs. assignee” comparisons and confusion logic.
• scoring/ (models.py, tasks.py) – raw annotation scoring (IOU, label, attribute).
• utils/ – helper functions (geometry, matching, plotting).
• __version__.py, __init__.py – package metadata.
c. External Dependencies
• Dataloop Platform (API server + item/annotation storage)
• Python libs: dtlpy SDK, numpy, shapely, OpenCV, requests, etc.
d. Configuration & CI
• dataloop.json – Dataloop project credentials/config.
• requirements.txt, setup.py – dependency declaration and packaging.
• GitHub workflows (.github/workflows) – CI tests, build, publish, tag.
e. Tests & Assets
• tests/ – unit and integration tests using stored JSON annotation assets.
• docs/ – design and function documentation (metrics.ipynb, dtlpymetrics_fxns.md).
Relationships and Data Flows
• CLI Runner → imports dtlpymetrics → loads dataloop.json → authenticates to Dataloop API.
• dtlpy_scores.py → calls scoring.tasks (raw scores) and evaluating.tasks (confusion, overall).
• scoring.tasks → use utils (matching, geometry) → calculate annotation-level scores.
• evaluating.tasks → aggregate raw scores → compute user_confusion, label_confusion, item_overall.
• dtlpy_scores uploads computed scores back to Dataloop via dtlpy SDK.
• In-pipeline Custom Nodes (tasks.py) are invoked by Dataloop when a task’s annotations complete, mirroring CLI flow but event-driven by the pipeline engine.
Architectural Patterns & Design Principles
• Modular separation: “scoring” vs. “evaluating” vs. “utils” layers.
• Pluggable pipeline design: CLI and SDK-driven nodes share core logic.
• Single responsibility: each task.py handles one type of score.
• Config-driven: dataloop.json and environment variables drive authentication and pipeline behavior.
• Testable: assets under tests/ simulate real annotation payloads; enables CI pipelines to verify scoring outputs.