How sarick-story/story-llm-influence-experiment works
Step 1. Project Type & Purpose:
• The project is a research/analysis framework centered around small language model training (TinyLlama-1b) and influence evaluation.
• Key operations are training a model; computing “influence factors” and “scores” via Kronfluence; and running evaluations (both custom and using standard benchmarks via OLMES).
Step 2. Analyze the File Structure:
• Top-level Files: Files like README.md, config.yaml, main.py, run.py, and run_full_pipeline.sh indicate central orchestration and configuration.
• Modular Organization: The modules directory houses several submodules:
– modules/training: Contains model training scripts.
– modules/analysis: Further divided into “factors” and “scores” for computing and inspecting influence metrics.
– modules/evaluation: Contains subdirectories (custom, deepeval, olmes, reporting) for running evaluations and reporting the results.
– modules/utils: Contains utility functions (e.g., wandb_utils) for logging and other auxiliary functions.
• Results Directories: The “results” folder indicates how outputs and reports are organized.
• Configuration & Prompts: Files like config.yaml and prompts.json determine how the system operates and what inputs/queries are evaluated.
Step 3. Insights from the README:
• The README provides a roadmap: training, influence analysis (factors, scores, inspections) and evaluations.
• It explains usage patterns via command-line wrappers and detailed instructions regarding hardware (e.g., NVIDIA A100 GPU), session management (tmux), and monitoring tools (nvitop).
• Centralized configuration in config.yaml means that the system’s behavior is dictated by unified configuration parameters.
a. Main Components:
– Configuration Component: config.yaml (centralized config for model paths, dataset, output directories, and evaluation settings).
– Training Module: modules/training (run by main.py or run.py for training operations).
– Influence Analysis Module:
• Factor Computation: modules/analysis/factors (compute.py, inspect.py, task.py).
• Scores Computation: modules/analysis/scores (compute.py, inspect.py).
– Evaluation Module:
• Custom Evaluation: modules/evaluation/custom (compare.py, generate_answers.py).
• OLMES Evaluation: modules/evaluation/olmes (olmes_runner.py).
• Deep Evaluation (deepeval): modules/evaluation/deepeval (deepeval_runner.py).
• Reporting: modules/evaluation/reporting (combine.py).
– Utility Functions: modules/utils (wandb_utils.py) for logging, monitoring, etc.
– External Dependencies: Kronfluence and OLMES, which interface with various evaluation and analysis functions.
– Orchestration: main.py and run.py as centralized entry points that invoke these modules.
b. Relationships & Interactions:
– The orchestrator (main.py or run.py) reads the configuration (config.yaml) to decide what workflow to run (training, analysis, evaluation, or full pipeline).
– The training module produces a trained TinyLlama model whose outputs are then fed (via configured scripts) into the analysis module.
– Within the analysis, ‘factors’ abstraction is responsible for calculating influence metrics, and ‘scores’ are computed subsequently.
– The evaluation module takes the results of both training and analysis, running custom metrics (e.g., BLEU, ROUGE) and standardized benchmarks via OLMES.
– The reporting module combines results from both evaluation methodologies.
– Utility modules support by handling logging (potentially upping integration with experiment monitoring tools like wandb).
c. Architectural Patterns & Design Principles:
– Modular Separation: The system is split into distinct modules, each responsible for a specific stage of the pipeline (training, analysis, evaluation).
– Centralized Configuration: By using config.yaml it enforces a single source of truth.
– Extendability: Separate directories for custom and standardized metrics (OLMES, deepeval) make it easy to add or modify evaluation steps.
– Pipeline Orchestration: The main driver (main.py) acts as a controller that coordinates between various stages using command line arguments.
d. Relevant Technologies and Libraries:
– PyTorch and Transformers for deep learning & language model training.
– Datasets (via Hugging Face) for managing training data.
– Kronfluence, used for influence analysis.
– OLMES for standardized evaluation benchmarks.
– Additional tools: tmux for session management, nvitop for GPU monitoring, and wandb for tracking experiments.
– “Training” (modules/training, run.py commands)
– “Influence Analysis” with sub-components “Factors” and “Scores”
– “Evaluation” with branches for “Custom Evaluation” and “OLMES Evaluation”
• Data flow starting from configuration input (config.yaml and prompts.json) to training, then to influence analysis, and finally aggregating results in the evaluation module.
• Use clear labels for each module (e.g., “Training Module”, “Influence Analysis – Factors”, “Influence Analysis – Scores”, “Evaluation – Custom”, “Evaluation – OLMES”).
– From “Configuration (config.yaml)” to every major module, signifying configuration dependency.
– From “Training Module” output (trained model) to “Influence Analysis”.
– From “Influence Analysis” outputs to “Evaluation Modules”.
• Text annotations to capture important design principles (e.g., “Centralized Configuration”, “Modular Pipeline”, “External Library Integration”).
Step 7. Final Tips: