How ddidacus/loco-llm works
LoCo-LMs fine-tunes and evaluates language models with neuro-symbolic losses intended to improve factual and logical consistency. The main workflow is a user-supplied configuration and constraint choice entering run.py, which assembles the model, constraint manager, and trainer; training combines model belief probabilities with symbolic constraint loss, while evaluation scores consistency and related metrics. Data preparation supports BeliefBank and ConceptNet; README also documents EntailmentBank evaluation. The alternative-dataset scripts and some trainer wiring are not sampled, so their internal call paths are left unspecified.
Main entry points: run.py Run entry point · trainer.py Trainer · model.py QA language model · eval.py Consistency metrics
Components 17
Run orchestration
- Experiment configs
configs - Run entry point
run.py
Fact and rule data
- BeliefBank data
dataset/beliefbank.py - ConceptNet data
dataset/conceptnet.py - Dataset primitives
dataset/dataset.py - ConceptNet extraction
scripts/get_conceptnet.py
Neuro-symbolic training
- QA language model
models/loco/model.py - Prompt formats
models/loco/prompts.py - Trainer
models/loco/trainer.py - Constraint manager
utils/constraints.py - Training utilities
utils/training.py
Evaluation
- Consistency metrics
utils/eval.py - EntailmentBank evaluation
scripts/experiments/eval_entailmentbank.py - BeliefBank evaluation
scripts/experiments/eval_beliefbank_binary.py
Other components
- Experiment user
- Transformers model source
- Weights & Biases
Connections 17
- Experiment user → Run entry point: starts run
- Run entry point → Experiment configs: reads settings
- Run entry point → QA language model: constructs model
- Run entry point → Constraint manager: constructs manager
- Run entry point → Trainer: starts task
- Run entry point → Weights & Biases: logs run
- QA language model → Transformers model source: loads model
- QA language model → Prompt formats: formats prompts
- Trainer → QA language model: queries beliefs
- Trainer → Constraint manager: computes loss
- Trainer → BeliefBank data: prepares data
- Trainer → Consistency metrics: scores results
- BeliefBank data → Dataset primitives: uses samples
- ConceptNet data → Dataset primitives: uses samples
- ConceptNet extraction → ConceptNet data: supports extraction
- EntailmentBank evaluation → Consistency metrics: evaluates consistency
- BeliefBank evaluation → Consistency metrics: evaluates consistency