# sarick-story/story-llm-influence-experiment architecture

> Architecture diagram of the GitHub repository sarick-story/story-llm-influence-experiment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarick-story/story-llm-influence-experiment
- Repository: https://github.com/sarick-story/story-llm-influence-experiment

## Overview

Step 1. Project Type & Purpose:
- The project is a research/analysis framework centered around small language model training (TinyLlama-1b) and influence evaluation.
- Key operations are training a model; computing “influence factors” and “scores” via Kronfluence; and running evaluations (both custom and using standard benchmarks via OLMES).

Step 2. Analyze the File Structure:
- Top-level Files: Files like README.md, config.yaml, main.py, run.py, and run_full_pipeline.sh indicate central orchestration and configuration.
- Modular Organization: The modules directory houses several submodules:
– modules/training: Contains model training scripts.
– modules/analysis: Further divided into “factors” and “scores” for computing and inspecting influence metrics.
– modules/evaluation: Contains subdirectories (custom, deepeval, olmes, reporting) for running evaluations and reporting the results.
– modules/utils: Contains utility functions (e.g., wandb_utils) for logging and other auxiliary functions.
- Results Directories: The “results” folder indicates how outputs and reports are organized.
- Configuration & Prompts: Files like config.yaml and prompts.json determine how the system operates and what inputs/queries are evaluated.

Step 3. Insights from the README:
- The README provides a roadmap: training, influence analysis (factors, scores, inspections) and evaluations.
- It explains usage patterns via command-line wrappers and detailed instructions regarding hardware (e.g., NVIDIA A100 GPU), session management (tmux), and monitoring tools (nvitop).
- Centralized configuration in config.yaml means that the system’s behavior is dictated by unified configuration parameters.

a. Main Components:
– Configuration Component: config.yaml (centralized config for model paths, dataset, output directories, and evaluation settings).
– Training Module: modules/training (run by main.py or run.py for training operations).
– Influence Analysis Module:
- Factor Computation: modules/analysis/factors (compute.py, inspect.py, task.py).
- Scores Computation: modules/analysis/scores (compute.py, inspect.py).
– Evaluation Module:
- Custom Evaluation: modules/evaluation/custom (compare.py, generate_answers.py).
- OLMES Evaluation: modules/evaluation/olmes (olmes_runner.py).
- Deep Evaluation (deepeval): modules/evaluation/deepeval (deepeval_runner.py).
- Reporting: modules/evaluation/reporting (combine.py).
– Utility Functions: modules/utils (wandb_utils.py) for logging, monitoring, etc.
– External Dependencies: Kronfluence and OLMES, which interface with various evaluation and analysis functions.
– Orchestration: main.py and run.py as centralized entry points that invoke these modules.

b. Relationships & Interactions:
– The orchestrator (main.py or run.py) reads the configuration (config.yaml) to decide what workflow to run (training, analysis, evaluation, or full pipeline).
– The training module produces a trained TinyLlama model whose outputs are then fed (via configured scripts) into the analysis module.
– Within the analysis, ‘factors’ abstraction is responsible for calculating influence metrics, and ‘scores’ are computed subsequently.
– The evaluation module takes the results of both training and analysis, running custom metrics (e.g., BLEU, ROUGE) and standardized benchmarks via OLMES.
– The reporting module combines results from both evaluation methodologies.
– Utility modules support by handling logging (potentially upping integration with experiment monitoring tools like wandb).

c. Architectural Patterns & Design Principles:
– Modular Separation: The system is split into distinct modules, each responsible for a specific stage of the pipeline (training, analysis, evaluation).
– Centralized Configuration: By using config.yaml it enforces a single source of truth.
– Extendability: Separate directories for custom and standardized metrics (OLMES, deepeval) make it easy to add or modify evaluation steps.
– Pipeline Orchestration: The main driver (main.py) acts as a controller that coordinates between various stages using command line arguments.

d. Relevant Technologies and Libraries:
– PyTorch and Transformers for deep learning & language model training.
– Datasets (via Hugging Face) for managing training data.
– Kronfluence, used for influence analysis.
– OLMES for standardized evaluation benchmarks.
– Additional tools: tmux for session management, nvitop for GPU monitoring, and wandb for tracking experiments.

– “Training” (modules/training, run.py commands)
– “Influence Analysis” with sub-components “Factors” and “Scores”
– “Evaluation” with branches for “Custom Evaluation” and “OLMES Evaluation”
- Data flow starting from configuration input (config.yaml and prompts.json) to training, then to influence analysis, and finally aggregating results in the evaluation module.

- Use clear labels for each module (e.g., “Training Module”, “Influence Analysis – Factors”, “Influence Analysis – Scores”, “Evaluation – Custom”, “Evaluation – OLMES”).
– From “Configuration (config.yaml)” to every major module, signifying configuration dependency.
– From “Training Module” output (trained model) to “Influence Analysis”.
– From “Influence Analysis” outputs to “Evaluation Modules”.
- Text annotations to capture important design principles (e.g., “Centralized Configuration”, “Modular Pipeline”, “External Library Integration”).

Step 7. Final Tips:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph: Configuration
    subgraph "Configuration"
        config["config.yaml"]:::configStyle
        prompts["prompts.json"]:::configStyle
    end

    %% Subgraph: Orchestration
    subgraph "Orchestration"
        mainOrch["main.py"]:::orchestratorStyle
        runOrch["run.py"]:::orchestratorStyle
    end

    %% Subgraph: Training Module
    subgraph "Training Module"
        training["Training Module"]:::trainingStyle
    end

    %% Subgraph: Influence Analysis
    subgraph "Influence Analysis"
        factors["Influence Analysis - Factors"]:::analysisStyle
        scores["Influence Analysis - Scores"]:::analysisStyle
    end

    %% Subgraph: Evaluation
    subgraph "Evaluation"
        custom["Evaluation - Custom"]:::evaluationStyle
        olmes["Evaluation - OLMES"]:::evaluationStyle
        deep["Evaluation - Deep"]:::evaluationStyle
        reporting["Evaluation - Reporting"]:::evaluationStyle
    end

    %% Utility Functions and External Dependencies
    utility["Utility Functions"]:::utilityStyle
    externalKron["Kronfluence"]:::externalStyle
    externalOlmes["OLMES"]:::externalStyle

    %% Connections from Configuration to Orchestration
    config --> mainOrch
    prompts --> mainOrch
    config --> runOrch
    prompts --> runOrch

    %% Connections from Orchestration to Training Module
    mainOrch --> training
    runOrch  --> training

    %% Connection from Training Module to Influence Analysis
    training --> factors
    training --> scores

    %% Connections from Influence Analysis to Evaluation Modules
    factors --> custom
    factors --> olmes
    factors --> deep
    scores   --> custom
    scores   --> olmes
    scores   --> deep

    %% Aggregation in Evaluation Module
    custom --> reporting
    olmes  --> reporting
    deep   --> reporting

    %% External Dependencies
    externalKron --> factors
    externalKron --> scores
    externalOlmes --> olmes

    %% Utility Functions dotted support arrows
    training -.-> utility
    factors  -.-> utility
    scores   -.-> utility
    custom   -.-> utility
    olmes    -.-> utility
    deep     -.-> utility
    reporting -.-> utility
    mainOrch -.-> utility
    runOrch  -.-> utility

    %% Click Events
    click config "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/config.yaml"
    click prompts "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/prompts.json"
    click mainOrch "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/main.py"
    click runOrch "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/run.py"
    click training "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/modules/training/train.py"
    click factors "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/analysis/factors"
    click scores "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/analysis/scores"
    click custom "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/evaluation/custom"
    click olmes "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/evaluation/olmes"
    click deep "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/evaluation/deepeval"
    click reporting "https://github.com/sarick-story/story-llm-influence-experiment/tree/master/modules/evaluation/reporting"
    click utility "https://github.com/sarick-story/story-llm-influence-experiment/blob/master/modules/utils/wandb_utils.py"

    %% Styles
    classDef configStyle fill:#FFEB3B,stroke:#F57F17,stroke-width:2px;
    classDef orchestratorStyle fill:#8BC34A,stroke:#33691E,stroke-width:2px;
    classDef trainingStyle fill:#64B5F6,stroke:#0D47A1,stroke-width:2px;
    classDef analysisStyle fill:#E1BEE7,stroke:#6A1B9A,stroke-width:2px;
    classDef evaluationStyle fill:#FFCDD2,stroke:#B71C1C,stroke-width:2px;
    classDef utilityStyle fill:#CFD8DC,stroke:#37474F,stroke-width:2px;
    classDef externalStyle fill:#D7CCC8,stroke:#4E342E,stroke-width:2px;
```
