# nasa-impact/evalem architecture

> Architecture diagram of the GitHub repository nasa-impact/evalem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nasa-impact/evalem
- Repository: https://github.com/nasa-impact/evalem

## Overview

##### Project type and purpose
- This is a Python library (“evalem”) for evaluating machine-learning pipelines, with a core framework plus domain-specific extensions for computer vision (cv) and natural language processing (nlp).
- It provides abstractions for metrics, evaluators, model-wrappers, and end-to-end evaluation pipelines.

##### File structure
- Top-level:
- evalem/       ← main package
- notebooks/    ← exploratory notebooks
- tests/        ← unit tests
- setup.py, pyproject.toml, requirements.txt
- Inside evalem/:
- _base/        ← core abstractions (abc, evaluators, metrics, models, pipelines, structures)
- api.py        ← top-level entrypoint functions (e.g., SimpleEvaluationPipeline)
- cv/           ← CV-specific metrics, evaluators, pipelines, models
- nlp/          ← NLP-specific metrics, evaluators, models, structures
- misc/         ← shared utilities
- Configuration: pytest.ini, tox.ini, GitHub workflows for linting/tests, pre-commit config.

##### Readme insights
- Core DTOs in _base.structures (PredictionDTO, ReferenceDTO, MetricResult).
- Metric hierarchy: Metric (base) → BasicMetric (precision, recall, etc.) → NLPMetric → SemanticMetric.
- Evaluator: container for running multiple metrics at once.
- Model wrappers for Huggingface models (HFWrapper, HFLMWrapper, HFPipelineWrapper, QA and TextClassification wrappers).
- SimpleEvaluationPipeline: orchestrates model wrappers + evaluators for end-to-end runs.

a. Components:
1. Client / User code (scripts, notebooks, CLI)
2. API layer (evalem/api.py: e.g., SimpleEvaluationPipeline)
3. Pipeline orchestrator (SimpleEvaluationPipeline)
4. Model-wrapper layer
– Generic base wrapper (ModelWrapper)
– HF-specific wrappers (HFLMWrapper, HFPipelineWrapper, QA & TextClassification wrappers)
5. Evaluator layer
– Base Evaluator (_base.evaluators.Evaluator)
– Domain-specific evaluators (cv, nlp subpackages)
6. Metric layer
– Base Metric classes (_base.metrics.Metric, BasicMetric)
– Domain-specific metrics (cv.metrics._base, nlp.metrics.*)
7. DTO / Data structures (_base.structures: PredictionDTO, ReferenceDTO, MetricResult)
8. External service / dependencies
– Huggingface Hub & Transformers
– PyTorch/CUDA/MPS (for device execution)
b. Relationships & data flow:
- Client code calls API → instantiates SimpleEvaluationPipeline with:
– a wrapped model (inputs → predictions)
– one or more Evaluator instances (each holding multiple Metric objects)
- Pipeline.run(inputs, references):
1. Inputs pass to Model Wrapper → executes underlying model (HF model/pipeline) → returns predictions (PredictionDTOs)
2. Evaluators receive (predictions, references):
– Each Metric transforms DTOs internally based on evaluation mode (SRSP, MRMP, etc.)
– Metrics compute scores → return MetricResult
3. Pipeline aggregates all MetricResults → final output
- Tests & notebooks consume same API to verify behavior and illustrate usage.
c. Architectural patterns / design principles:
– Layered architecture: API → Orchestrator → Model-wrapper → Evaluator → Metric → DTOs
– Strategy pattern: Metric implementations plugged into Evaluator
– Adapter pattern: ModelWrapper adapts arbitrary models to a uniform interface
– Separation of concerns: core vs domain-specific (cv, nlp) extensions
d. Key technologies/frameworks:
– Python packaging (hatch, setuptools)
– pytest for testing
– Huggingface transformers/pipelines
– PyTorch/CUDA/MPS support

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    Client["Client Code / Notebooks"]:::client

    %% API Layer
    API["evalem/api.py\nSimpleEvaluationPipeline"]:::core

    %% Core Orchestrator
    Pipeline["evalem/_base/pipelines.py\nBasePipeline"]:::core

    %% Model Wrapper Layer
    ModelWrapperBase["evalem/_base/models.py\nModelWrapper (base)"]:::core

    %% Evaluator Layer
    EvaluatorBase["evalem/_base/evaluators.py\nEvaluator (base)"]:::core

    %% Metric Layer
    MetricsBase["evalem/_base/metrics.py\nMetric & BasicMetrics"]:::core

    %% DTO Layer
    DTOs["evalem/_base/structures.py\nDTOs (PredictionDTO, ReferenceDTO, MetricResult)"]:::core

    %% Shared Utils
    MiscUtils["evalem/misc/utils.py\nShared Utilities"]:::core

    %% CV Domain Extensions
    subgraph "Computer Vision Extensions" 
        direction TB
        CVEvaluator["evalem/cv/evaluators/_base.py\nCVEvaluator"]:::domain
        CVMetrics["evalem/cv/metrics/_base.py\nCVMetrics"]:::domain
        CVPipelines["evalem/cv/pipelines/__init__.py\nCV Pipelines"]:::domain
        CVModels["evalem/cv/models/__init__.py\nCV Models"]:::domain
        CVDatasets["evalem/cv/misc/datasets.py\nCV Datasets"]:::domain
    end

    %% NLP Domain Extensions
    subgraph "NLP Extensions"
        direction TB
        NLPEvaluatorBase["evalem/nlp/evaluators/_base.py\nNLPEvaluatorBase"]:::domain
        NLPBasicsEvaluator["evalem/nlp/evaluators/basics.py\nBasic Evaluators"]:::domain
        NLPDummyEvaluator["evalem/nlp/evaluators/dummy.py\nDummy Evaluator"]:::domain
        NLPMetricBase["evalem/nlp/metrics/_base.py\nNLPMetricBase"]:::domain
        NLPBasicsMetrics["evalem/nlp/metrics/basics.py\nBasic Metrics"]:::domain
        NLPDummyMetrics["evalem/nlp/metrics/dummy.py\nDummy Metrics"]:::domain
        NLPLLMMetrics["evalem/nlp/metrics/llm.py\nLLM-Based Metrics"]:::domain
        NLPSemanticMetrics["evalem/nlp/metrics/semantics.py\nSemantic Metrics"]:::domain
        NLPModelsBase["evalem/nlp/models/_base.py\nNLPModels Base"]:::domain
        NLPModelsDefaults["evalem/nlp/models/defaults.py\nDefault HF Wrappers"]:::domain
        NLPDatasets["evalem/nlp/misc/datasets.py\nNLP Datasets"]:::domain
        NLPMiscUtils["evalem/nlp/misc/utils.py\nNLP Misc Utils"]:::domain
        NLPStructures["evalem/nlp/structures.py\nNLP Structures"]:::domain
    end

    %% External Dependencies
    Huggingface[(Huggingface Hub / Transformers)]:::external
    Device[(Device:\nCPU / GPU / MPS)]:::external

    %% Connections
    Client -->|"calls"| API
    API -->|"instantiates"| Pipeline
    Pipeline -->|"uses"| ModelWrapperBase
    Pipeline -->|"uses"| EvaluatorBase
    Pipeline -->|"uses"| DTOs
    ModelWrapperBase -->|"adapts to"| Huggingface
    ModelWrapperBase -->|"executes on"| Device
    EvaluatorBase -->|"holds metrics"| MetricsBase
    DTOs --> MetricsBase

    %% Domain plugs
    CVEvaluator -->|"extends"| EvaluatorBase
    CVMetrics -->|"extends"| MetricsBase
    CVPipelines -->|"builds on"| Pipeline
    CVModels -->|"extends"| ModelWrapperBase
    CVDatasets -->|"uses"| MiscUtils

    NLPEvaluatorBase -->|"extends"| EvaluatorBase
    NLPBasicsEvaluator -->|"implements"| NLPEvaluatorBase
    NLPDummyEvaluator -->|"implements"| NLPEvaluatorBase
    NLPMetricBase -->|"extends"| MetricsBase
    NLPBasicsMetrics -->|"implements"| NLPMetricBase
    NLPDummyMetrics -->|"implements"| NLPMetricBase
    NLPLLMMetrics -->|"implements"| NLPMetricBase
    NLPSemanticMetrics -->|"implements"| NLPMetricBase
    NLPModelsBase -->|"extends"| ModelWrapperBase
    NLPModelsDefaults -->|"implements"| NLPModelsBase
    NLPDatasets -->|"uses"| MiscUtils
    NLPMiscUtils -->|"shared with"| MiscUtils
    NLPStructures -->|"extends"| DTOs

    %% Click Events
    click API "https://github.com/nasa-impact/evalem/blob/develop/evalem/api.py"
    click ABC "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/abc.py"
    click Pipeline "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/pipelines.py"
    click EvaluatorBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/evaluators.py"
    click MetricsBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/metrics.py"
    click ModelWrapperBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/models.py"
    click DTOs "https://github.com/nasa-impact/evalem/blob/develop/evalem/_base/structures.py"
    click CVEvaluator "https://github.com/nasa-impact/evalem/blob/develop/evalem/cv/evaluators/_base.py"
    click CVMetrics "https://github.com/nasa-impact/evalem/blob/develop/evalem/cv/metrics/_base.py"
    click CVPipelines "https://github.com/nasa-impact/evalem/blob/develop/evalem/cv/pipelines/__init__.py"
    click CVModels "https://github.com/nasa-impact/evalem/blob/develop/evalem/cv/models/__init__.py"
    click CVDatasets "https://github.com/nasa-impact/evalem/blob/develop/evalem/cv/misc/datasets.py"
    click NLPEvaluatorBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/evaluators/_base.py"
    click NLPBasicsEvaluator "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/evaluators/basics.py"
    click NLPDummyEvaluator "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/evaluators/dummy.py"
    click NLPMetricBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/metrics/_base.py"
    click NLPBasicsMetrics "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/metrics/basics.py"
    click NLPDummyMetrics "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/metrics/dummy.py"
    click NLPLLMMetrics "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/metrics/llm.py"
    click NLPSemanticMetrics "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/metrics/semantics.py"
    click NLPModelsBase "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/models/_base.py"
    click NLPModelsDefaults "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/models/defaults.py"
    click NLPDatasets "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/misc/datasets.py"
    click NLPMiscUtils "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/misc/utils.py"
    click NLPStructures "https://github.com/nasa-impact/evalem/blob/develop/evalem/nlp/structures.py"
    click MiscUtils "https://github.com/nasa-impact/evalem/blob/develop/evalem/misc/utils.py"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,color:#1B2631;
    classDef domain fill:#ABEBC6,stroke:#186A3B,color:#1B2631;
    classDef external fill:#F9E79F,stroke:#7D6608,color:#1B2631;
    classDef client fill:#D5D8DC,stroke:#566573,color:#1B2631;
```
