# ivanvmoreno/correctness-model-internals architecture

> Architecture diagram of the GitHub repository ivanvmoreno/correctness-model-internals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ivanvmoreno/correctness-model-internals
- Repository: https://github.com/ivanvmoreno/correctness-model-internals

## Overview

##### Project type and purpose
- This is a research‐oriented data‐processing pipeline (“full stack” in the data‐science sense) for probing large language model (LLM) internals.
- It ingests raw question/answer datasets, runs questions through LLMs, captures intermediate activations, trains linear probes to predict correctness, and evaluates probe performance.

##### High‐level components
- Orchestration & environment
– Docker container (docker/Dockerfile + scripts) provides reproducible compute.
– DVC (dvc.yaml, dvc.lock) and Taskfile.yaml manage the multi‐stage pipeline and data versioning.
- Data stores
– Raw input datasets (datasets/raw/*).
– Custom/derived CSVs (datasets/custom/*).
– Captured activations and embeddings (artifacts produced by pipeline stages).
– Experiment outputs (experiment_results/*, evaluations_revisited/*).
- Pipeline stages (src/stages/*.py)
1. download_hf: fetch HF datasets/models
2. format_datasets: normalize raw data into CSV/parquet
3. generate_answers: run LLM on questions
4. capture_activations: extract hidden‐layer tensors mid‐inference
5. postprocess_activations: aggregate or reshape tensors
6. generate_embeddings: convert activations into fixed‐size embeddings
7. train_classifiers & train_embedding_assessors: fit linear probes
8. collect_classification_stats: compute predictions vs. ground truth
9. evaluate_answers: compare against baselines
- Core libraries (src/*.py)
– classifying/: activation handlers, direction calculators, classifier utilities
– llm_judge.py & model.py: LLM interface and model wrapper
– data.py, utils/*: configuration, logging, data I/O, metrics
– visualisations/: functions for embedding‐probe result plots
- Analysis & reporting
– Jupyter notebooks (notebooks/*) for exploratory analysis, best‐layer search, plotting.
– Custom scripts for “truth_analysis_arnau” and “verbalized_confidence” experiments.

##### Interactions & data flow
1. User invokes Taskfile or DVC:
‣ Docker container spins up; DVC fetches raw data.
2. format_datasets reads datasets/raw → writes standardized CSV/parquet.
3. generate_answers sends questions to HF or RunPod pods → writes model outputs.
4. capture_activations hooks into model runs → stores layer activations.
5. postprocess_activations & generate_embeddings convert activations to vectors.
6. train_classifiers & train_embedding_assessors read embeddings + labels → produce probe weights and prediction scores.
7. collect_classification_stats & evaluate_answers compute accuracy, ROC AUC, directional scores → saved under experiment_results and evaluations_revisited.

##### Architectural patterns & principles
- Pipeline as directed acyclic graph (DAG) managed by DVC.
- Modular stages: each “src/stages” script is a self‐contained step that reads inputs and writes outputs.
- Separation of concerns:
– Data ingestion vs. preprocessing vs. model inference vs. feature extraction vs. modeling vs. evaluation vs. visualization.
- Configuration in params.yaml and src/utils/config.py drives reproducibility and parameter sweeps.
- Containerization (Docker) ensures environment parity.

##### Technologies & external services
- Python 3.x, pyproject.toml, requirements-cpu.txt for dependencies.
- DVC for data pipeline orchestration and artifact versioning.
- Docker for containerized execution.
- Hugging Face and/or RunPod for LLM inference.
- Jupyter notebooks for ad‐hoc analysis.
- Linear classifiers (scikit-learn or similar) for probes.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Environment & Orchestration
    subgraph "Environment & Orchestration"
        style Env fill:#ADD8E6,stroke:#000
        EnvDocker["Dockerfile & Scripts"]:::orchestration
        EnvDVC["DVC (dvc.yaml, dvc.lock)"]:::orchestration
        EnvTask["Taskfile.yaml"]:::orchestration
    end

    %% Data Stores
    subgraph "Data Stores"
        style Store fill:#90EE90,stroke:#000
        RawData["datasets/raw/"]:::storage
        CustomData["datasets/custom/"]:::storage
        Activations["artifacts/activations & embeddings"]:::storage
        ExpResults["experiment_results/"]:::storage
        EvalRevisit["evaluations_revisited/"]:::storage
    end

    %% Pipeline Stages
    subgraph "Pipeline Stages"
        style Stages fill:#FFA500,stroke:#000
        S1["download_hf"]:::stage
        S2["format_datasets"]:::stage
        S3["generate_answers"]:::stage
        S4["capture_activations"]:::stage
        S5["postprocess_activations"]:::stage
        S6["generate_embeddings"]:::stage
        S7["train_classifiers"]:::stage
        S8["train_embedding_assessors"]:::stage
        S9["collect_classification_stats"]:::stage
        S10["evaluate_answers"]:::stage
    end

    %% Core Libraries
    subgraph "Core Libraries"
        style Core fill:#FFD580,stroke:#000
        CoreLLM["llm_judge.py & model.py"]:::core
        CoreData["data.py"]:::core
        CoreConfig["src/utils/config.py & params.yaml"]:::core
        CoreLogging["src/utils/logging.py, metrics.py"]:::core
        CoreUtils["src/utils/utils.py & data.py"]:::core
        CoreClass["src/classifying modules"]:::core
        CoreVis["src/visualisations"]:::core
    end

    %% Analysis & Reporting
    subgraph "Analysis & Reporting"
        style Analysis fill:#DDA0DD,stroke:#000
        NB["notebooks/"]:::analysis
        NBBest["notebooks/best_layer_finding/"]:::analysis
        TA["truth_analysis_arnau/"]:::analysis
        VC["verbalized_confidence/"]:::analysis
        AuxScript["src/scripts/train_embedding_assessors_combinations.py"]:::analysis
    end

    %% External Services
    subgraph "External Services"
        style Ext fill:#D3D3D3,stroke:#000
        HF["Hugging Face API"]:::external
        RP["RunPod"]:::external
    end

    %% Connections - Orchestration to Pipeline
    EnvDocker -->|"start"| S1
    EnvDVC -->|"DAG start"| S1
    EnvTask -->|"runs pipeline"| S1

    %% Pipeline flow
    S1 -->|"raw datasets"| RawData
    RawData -->|"reads raw"| S2
    S2 -->|"CSV/Parquet"| CustomData
    CustomData -->|"questions"| S3
    S3 -->|"model calls"| HF
    S3 -->|"or pods"| RP
    S3 -->|"answers"| Activations
    S3 --> S4
    S4 -->|"activations"| Activations
    Activations -->|"tensors"| S5
    S5 -->|"reshaped"| Activations
    Activations -->|"vectors"| S6
    S6 -->|"embeddings"| Activations
    Activations --> S7
    Activations --> S8
    S7 -->|"probe weights & scores"| ExpResults
    S8 -->|"probe weights & scores"| ExpResults
    ExpResults --> S9
    ExpResults --> S10
    S9 -->|"stats"| EvalRevisit
    S10 -->|"metrics"| EvalRevisit

    %% Analysis reads results
    EvalRevisit -->|"reads metrics"| NB
    EvalRevisit --> NBBest
    EvalRevisit --> TA
    EvalRevisit --> VC
    NB --> AuxScript

    %% Core libraries usage
    S1 --> CoreLLM
    S3 --> CoreLLM
    S4 --> CoreClass
    S5 --> CoreUtils
    S6 --> CoreVis
    S7 --> CoreClass
    S8 --> CoreClass
    S9 --> CoreData
    S10 --> CoreLogging
    S2 --> CoreData

    %% Click Events
    click EnvDocker "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/docker/Dockerfile"
    click EnvDocker "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/docker/scripts/start.sh"
    click EnvDocker "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/docker/scripts/post_start.sh"
    click EnvDVC "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/dvc.yaml"
    click EnvDVC "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/dvc.lock"
    click EnvTask "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/Taskfile.yaml"
    click RawData "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/datasets/raw/"
    click CustomData "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/datasets/custom/"
    click ExpResults "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/experiment_results/"
    click EvalRevisit "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/evaluations_revisited/"
    click S1 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/download_hf.py"
    click S2 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/format_datasets.py"
    click S3 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/generate_answers.py"
    click S4 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/capture_activations.py"
    click S5 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/postprocess_activations.py"
    click S6 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/generate_embeddings.py"
    click S7 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/train_classifiers.py"
    click S8 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/train_embedding_assessors.py"
    click S9 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/collect_classification_stats.py"
    click S10 "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/stages/evaluate_answers.py"
    click CoreLLM "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/llm_judge.py"
    click CoreLLM "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/model.py"
    click CoreData "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/data.py"
    click CoreConfig "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/utils/config.py"
    click CoreConfig "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/params.yaml"
    click CoreLogging "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/utils/logging.py"
    click CoreLogging "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/utils/metrics.py"
    click CoreUtils "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/utils/utils.py"
    click CoreUtils "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/utils/data.py"
    click CoreClass "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/classifying/activations_handler.py"
    click CoreClass "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/classifying/direction_calculator.py"
    click CoreClass "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/classifying/classification_utils.py"
    click CoreVis "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/visualisations/embedding_assessors.py"
    click CoreVis "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/visualisations/utils.py"
    click NB "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/notebooks/"
    click NBBest "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/notebooks/best_layer_finding/"
    click TA "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/truth_analysis_arnau/"
    click VC "https://github.com/ivanvmoreno/correctness-model-internals/tree/main/verbalized_confidence/"
    click AuxScript "https://github.com/ivanvmoreno/correctness-model-internals/blob/main/src/scripts/train_embedding_assessors_combinations.py"

    %% Styles
    classDef orchestration fill:#ADD8E6,stroke:#000
    classDef storage fill:#90EE90,stroke:#000
    classDef stage fill:#FFA500,stroke:#000
    classDef core fill:#FFD580,stroke:#000
    classDef analysis fill:#DDA0DD,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
```
