How sepehrilami/lfb works
• Data Sources and Datasets:
• Directories such as /data (with subdirectories like context and game prompts), /new contexts, and /factory_ready.json (original fine-tuning dataset in LLaMafactory syntax).
• Specialized datasets for different experiments (in-sample, out-of-sample, public good game experimental data from directories /ood_game, /oos_context, /pgg).
• Experiment Orchestration and Model Pipeline:
• Main experiment execution is orchestrated by mainloop.py and the fine-tuning run from webui.ipynb.
• The fine-tuned model is housed in the /l2tune directory, complete with its adapter configurations and prebuilt model weights.
• Reference the usage of external models (LLaMa2-7b and 70b from Huggingface) and the local fine-tuned model.
• Analysis and Visualization:
• Notebooks directory houses various Jupyter notebooks (e.g., in-sample-context-analysis.ipynb, ood_game_analysis.ipynb, oos_analysis.ipynb, pgg_analysis.ipynb) that perform comparative analysis and visualization.
• The visualization outputs and figures are stored in directories such as /new_figs, /ood_game_figs, /oos_context_figs, and /pgg_figs.
• Results Storage:
• Components like /results, /results_llama3, and associated data summary files (e.g., .csv, .pkl, .txt) which record experiment metrics and aggregated outputs.
Component Interactions and Data Flow
• The data layer (raw data files and context/game prompts) flows into the experiment orchestration layer where models are either fine-tuned or evaluated.
• How input datasets from /data and /new contexts are fed to the fine-tuning procedure.
• The transitions from the fine-tuning phase (executed in mainloop.py and webui.ipynb) to benchmark evaluation using both standard LLaMa2 models (fetched via Huggingface) and the fine-tuned model.
• How experimental outputs are stored in the /results-related directories, and then how these results are analyzed via the Jupyter notebooks.
• External dependencies (such as Groq platform requirements and Huggingface model APIs) as additional nodes connected to the experiment execution component.
Architectural Patterns and Design Principles
• Modular nature of the project:
• Data is clearly separated by type (context vs. game, in-sample versus out-of-sample).
• The experiment pipeline is divided into training/fine-tuning and evaluation, with distinct directories for execution scripts (mainloop.py and notebooks).
• Use of configuration files (e.g., requirements.txt for dependency management, adapter_config.json for model fine-tuning) that provide a reproducible environment.
• Design choice of using various file formats (.txt, .csv, .pkl, .ipynb) for different components, reflecting a data-driven and iterative research approach.