How causalnlp/cladder works
Project type and purpose
• This is an open-source Python package and dataset repository (“CLadder”) for generating, storing, and evaluating causal‐reasoning questions on language models.
• It provides:
• A dataset (cladder‐v1) of yes/no causal questions
• A generator to synthesize questions from causal graphs
• Evaluation scripts to run LLMs (GPT, LLaMA, Alpaca, etc.) on the data
• Statistics, visualizations, and utilities to analyze results
File structure
• Top‐level modules and scripts
• main.py – CLI entry point
• causalbenchmark/ – core package:
– generator.py – demo/full data generation
– eval/ – LLM evaluation scripts
– formatting.py – output formatting
– graphs/ – graph builders & plotting
– queries/ – query‐type implementations
– util.py – shared helpers
– verbal/ – story verbalizations
– test/ – unit tests
• assets/ – YAML story templates
• config/ – YAML configs for generation modes
• data/ – prebuilt JSON datasets and metadata
• demo-mini/ – sample demo data
• CI: .github/workflows/tests.yaml
• Packaging: setup.py, requirements.txt
README for architecture clues
• Describes two main workflows:
1. Data generation: uses assets + config to build cladder‐v1
2. Model evaluation: runs LLMs on questions and post‐processes answers
• Mentions template files, output formats, and evaluation metrics
Main components
a. Inputs
• Assets (assets/stories/*.yml)
• Config files (config/*.yaml)
• Optional external dataset (HuggingFace URL or local zip)
b. Data Generation
• Generator module (causalbenchmark/generator.py)
• Builds causal graphs (causalbenchmark/graphs/)
• Samples questions via templates (causalbenchmark/verbal/)
• Produces JSON files in data/ or demo-mini/
c. Evaluation Pipeline
• Eval scripts (causalbenchmark/eval/run_*.py)
• LLM APIs (OpenAI, LLaMA wrappers)
• Prompt assembly: background + given_info + question
• Raw outputs → Postprocessing (run_llama_postprocess.py)
d. Analysis & Visualization
• Formatting module (causalbenchmark/formatting.py)
• Stats & plotting (causalbenchmark/eval/data_stats_plot.py, graphs/)
• Output: tables, confusion matrices, metrics
e. Tests & CI
• Unit tests (causalbenchmark/test/)
• GitHub Actions CI
Relationships and data flow
• Assets + Config → Generator → Dataset (JSON)
• Dataset → Eval scripts → LLM endpoints → Raw model outputs
• Raw outputs → Postprocessing → Formatted results
• Formatted results → Stats & Graphs → Final evaluation artifacts
• Tests exercise all code modules; CI enforces quality
Architectural patterns and design principles
• Modular package structure by responsibility (generator, eval, graphs, queries)
• Configuration‐driven workflows (YAML files control behavior)
• Pluggable LLM backends (separate run_*.py for different models)
• Clear separation: data generation vs. evaluation vs. analysis
• Test coverage for each submodule
Technology highlights
• Python 3.x, setuptools packaging
• YAML parsing for configs & assets
• HuggingFace dataset support (optional)
• LLM API clients (OpenAI, llama_cpp, etc.)
• Matplotlib/Seaborn or Plotly for visualization
Implementation detail callouts
• Role of causalbenchmark/queries for implementing each question type
• Plotting modules in causalbenchmark/graphs for generating study figures
• Mention test harness under causalbenchmark/test to maintain code reliability