How nasa-impact/evalem works
Project type and purpose
• This is a Python library (“evalem”) for evaluating machine-learning pipelines, with a core framework plus domain-specific extensions for computer vision (cv) and natural language processing (nlp).
• It provides abstractions for metrics, evaluators, model-wrappers, and end-to-end evaluation pipelines.
File structure
• Top-level:
• evalem/ ← main package
• notebooks/ ← exploratory notebooks
• tests/ ← unit tests
• setup.py, pyproject.toml, requirements.txt
• Inside evalem/:
• _base/ ← core abstractions (abc, evaluators, metrics, models, pipelines, structures)
• api.py ← top-level entrypoint functions (e.g., SimpleEvaluationPipeline)
• cv/ ← CV-specific metrics, evaluators, pipelines, models
• nlp/ ← NLP-specific metrics, evaluators, models, structures
• misc/ ← shared utilities
• Configuration: pytest.ini, tox.ini, GitHub workflows for linting/tests, pre-commit config.
Readme insights
• Core DTOs in _base.structures (PredictionDTO, ReferenceDTO, MetricResult).
• Metric hierarchy: Metric (base) → BasicMetric (precision, recall, etc.) → NLPMetric → SemanticMetric.
• Evaluator: container for running multiple metrics at once.
• Model wrappers for Huggingface models (HFWrapper, HFLMWrapper, HFPipelineWrapper, QA and TextClassification wrappers).
• SimpleEvaluationPipeline: orchestrates model wrappers + evaluators for end-to-end runs.
a. Components:
1. Client / User code (scripts, notebooks, CLI)
2. API layer (evalem/api.py: e.g., SimpleEvaluationPipeline)
3. Pipeline orchestrator (SimpleEvaluationPipeline)
4. Model-wrapper layer
– Generic base wrapper (ModelWrapper)
– HF-specific wrappers (HFLMWrapper, HFPipelineWrapper, QA & TextClassification wrappers)
5. Evaluator layer
– Base Evaluator (_base.evaluators.Evaluator)
– Domain-specific evaluators (cv, nlp subpackages)
6. Metric layer
– Base Metric classes (_base.metrics.Metric, BasicMetric)
– Domain-specific metrics (cv.metrics._base, nlp.metrics.*)
7. DTO / Data structures (_base.structures: PredictionDTO, ReferenceDTO, MetricResult)
8. External service / dependencies
– Huggingface Hub & Transformers
– PyTorch/CUDA/MPS (for device execution)
b. Relationships & data flow:
• Client code calls API → instantiates SimpleEvaluationPipeline with:
– a wrapped model (inputs → predictions)
– one or more Evaluator instances (each holding multiple Metric objects)
• Pipeline.run(inputs, references):
1. Inputs pass to Model Wrapper → executes underlying model (HF model/pipeline) → returns predictions (PredictionDTOs)
2. Evaluators receive (predictions, references):
– Each Metric transforms DTOs internally based on evaluation mode (SRSP, MRMP, etc.)
– Metrics compute scores → return MetricResult
3. Pipeline aggregates all MetricResults → final output
• Tests & notebooks consume same API to verify behavior and illustrate usage.
c. Architectural patterns / design principles:
– Layered architecture: API → Orchestrator → Model-wrapper → Evaluator → Metric → DTOs
– Strategy pattern: Metric implementations plugged into Evaluator
– Adapter pattern: ModelWrapper adapts arbitrary models to a uniform interface
– Separation of concerns: core vs domain-specific (cv, nlp) extensions
d. Key technologies/frameworks:
– Python packaging (hatch, setuptools)
– pytest for testing
– Huggingface transformers/pipelines
– PyTorch/CUDA/MPS support