# cvs-health/langfair architecture

> Architecture diagram of the GitHub repository cvs-health/langfair, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cvs-health/langfair
- Repository: https://github.com/cvs-health/langfair

## Overview

Based on our analysis of the file tree and the README, LangFair is a Python library for running bias and fairness assessments of large language model use cases. It supports several different types of evaluations (toxicity, counterfactual, stereotype, classification, and recommendation), and it is designed so users can integrate their own prompts and LLMs via adapters (for example, using LangChain). The project is modular and organized into clearly separated components that handle generating responses, computing various metrics, and an “AutoEval” component that streamlines the whole process.

##### Main Components
- Core Library (langfair package): Include submodules like:
- auto (for the AutoEval class that automates multi-step evaluations)
- generator (for generating responses and counterfactual examples)
- metrics (with further subdivisions for classification, toxicity, counterfactual, stereotype, and recommendation)
- utils (providing helper functions, e.g. data loading)
- Data: Show the “data” directory which includes reference files like sample prompts or guidelines (DATA_COPYRIGHT, sample JSON lines)
- Examples: Represent the notebooks provided under examples for demonstrative use cases across different evaluation types.
- Tests: Represent the “tests” package that ensures the correctness of the library via unit tests and end-to-end evaluation demos.
- External Integrations: Emphasize integration with external libraries (e.g., LangChain for LLM connections, torch optionally for GPUs, and rate limiter services) which are used by the generator and evaluation components.

##### Relationships and Data Flow
- Start at the user’s side (client code using the library or demo notebooks) inputting custom prompts and parameters.
- How the client code interacts with:
- The Generator component (ResponseGenerator and CounterfactualGenerator) to obtain LLM outputs.
- These outputs then flow into different Metrics components (ToxicityMetrics, StereotypeMetrics, etc.) that compute bias and fairness scores.
- The AutoEval component wraps these steps to provide an all-in-one evaluation workflow.
- The computed metrics are then reported back to the client (and can be stored, visualized, or compared to benchmarks).
- How configuration and output data (log files, results from evaluations) allow for reproducibility and further analysis.

##### Architectural Patterns and Design Principles
- Modular Structure: The clear separation of functionality into submodules (auto, generator, metrics, and their sub-categories) is key. This follows a layered or modular pattern.

##### Technologies, Frameworks, and Tools
- Python as the primary programming language.
- Integration with LangChain and potential external rate limiters.
- Torch for GPU acceleration in certain metric computations (optional).
- GitHub workflows and configuration files (like pyproject.toml, pre-commit settings) as part of the development lifecycle.
- Notebooks shown in the examples folder serve as demonstration or “client” usage of these components.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Layer
    subgraph "Client Layer"
        Client("Client/Notebooks"):::client
    end

    %% Core Library Layer
    subgraph "LangFair Core"
        Core("Core Library"):::core
        AutoEval("AutoEval"):::core
        ResponseGen("ResponseGenerator"):::core
        CounterGen("CounterfactualGenerator"):::core
        ClassMetrics("ClassificationMetrics"):::core
        CounterMetrics("CounterfactualMetrics"):::core
        RecMetrics("RecommendationMetrics"):::core
        StereoMetrics("StereotypeMetrics"):::core
        ToxMetrics("ToxicityMetrics"):::core
        DataLoader("DataLoader"):::utility
    end

    %% Data and Testing Layer
    subgraph "Data & Testing"
        DataFiles("Data Files"):::utility
        TestSuite("Test Suite"):::utility
    end

    %% External Integrations Layer
    subgraph "External Integrations"
        LangChain("LangChain"):::external
        Torch("Torch"):::external
        RateLimiter("RateLimiter"):::external
    end

    %% Connections between Client and Core Modules
    Client -->|"inputPrompts"| AutoEval
    AutoEval -->|"invokesGenerator"| ResponseGen
    AutoEval -->|"orchestrates"| CounterGen

    %% Generator outputs flow into Metrics components
    ResponseGen -->|"LLMOutput"| ClassMetrics
    ResponseGen -->|"LLMOutput"| RecMetrics
    ResponseGen -->|"LLMOutput"| StereoMetrics
    ResponseGen -->|"LLMOutput"| ToxMetrics
    CounterGen -->|"LLMOutput"| CounterMetrics

    %% Metrics aggregation back to AutoEval and client
    ClassMetrics -->|"aggregatedMetrics"| AutoEval
    CounterMetrics -->|"aggregatedMetrics"| AutoEval
    RecMetrics -->|"aggregatedMetrics"| AutoEval
    StereoMetrics -->|"aggregatedMetrics"| AutoEval
    ToxMetrics -->|"aggregatedMetrics"| AutoEval
    AutoEval -->|"results"| Client

    %% External Integrations to Core Components
    LangChain -->|"adapterCalls"| ResponseGen
    RateLimiter -->|"controlsRequests"| ResponseGen
    Torch -->|"accelerates"| ToxMetrics
    Torch -->|"accelerates"| ClassMetrics

    %% Utility: Data Loading connection
    DataLoader -->|"loads"| DataFiles

    %% Testing validates Core Library
    TestSuite -->|"validates"| Core

    %% Legend Node (for informational purposes)
    Note1["Legend: Blue = Core Modules, Green = External, Orange = Utility/Testing, Purple = Client"]:::legend
    TestSuite --- Note1

    %% Click Events for component mapping
    click Core "https://github.com/cvs-health/langfair/tree/main/langfair/"
    click AutoEval "https://github.com/cvs-health/langfair/blob/main/langfair/auto/auto.py"
    click ResponseGen "https://github.com/cvs-health/langfair/blob/main/langfair/generator/generator.py"
    click CounterGen "https://github.com/cvs-health/langfair/blob/main/langfair/generator/counterfactual.py"
    click ClassMetrics "https://github.com/cvs-health/langfair/blob/main/langfair/metrics/classification/classification.py"
    click CounterMetrics "https://github.com/cvs-health/langfair/blob/main/langfair/metrics/counterfactual/counterfactual.py"
    click RecMetrics "https://github.com/cvs-health/langfair/blob/main/langfair/metrics/recommendation/recommendation.py"
    click StereoMetrics "https://github.com/cvs-health/langfair/blob/main/langfair/metrics/stereotype/stereotype.py"
    click ToxMetrics "https://github.com/cvs-health/langfair/blob/main/langfair/metrics/toxicity/toxicity.py"
    click DataLoader "https://github.com/cvs-health/langfair/blob/main/langfair/utils/dataloader.py"
    click DataFiles "https://github.com/cvs-health/langfair/tree/main/data/"
    click Client "https://github.com/cvs-health/langfair/tree/main/examples/"
    click TestSuite "https://github.com/cvs-health/langfair/tree/main/tests/"

    %% Class Definitions for styling
    classDef core fill:#add8e6,stroke:#00008b,stroke-width:2px;
    classDef external fill:#90ee90,stroke:#006400,stroke-width:2px;
    classDef utility fill:#ffcc99,stroke:#cc6600,stroke-width:2px;
    classDef client fill:#dda0dd,stroke:#800080,stroke-width:2px;
    classDef legend fill:#eeeeee,stroke:#333,stroke-dasharray: 5 5,stroke-width:2px;
```
