# causalnlp/cladder architecture

> Architecture diagram of the GitHub repository causalnlp/cladder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/causalnlp/cladder
- Repository: https://github.com/causalnlp/cladder

## Overview

##### Project type and purpose
- This is an open-source Python package and dataset repository (“CLadder”) for generating, storing, and evaluating causal‐reasoning questions on language models.
- It provides:
- A dataset (cladder‐v1) of yes/no causal questions
- A generator to synthesize questions from causal graphs
- Evaluation scripts to run LLMs (GPT, LLaMA, Alpaca, etc.) on the data
- Statistics, visualizations, and utilities to analyze results

##### File structure
- Top‐level modules and scripts
- main.py – CLI entry point
- causalbenchmark/ – core package:
– generator.py – demo/full data generation
– eval/ – LLM evaluation scripts
– formatting.py – output formatting
– graphs/ – graph builders & plotting
– queries/ – query‐type implementations
– util.py – shared helpers
– verbal/ – story verbalizations
– test/ – unit tests
- assets/ – YAML story templates
- config/ – YAML configs for generation modes
- data/ – prebuilt JSON datasets and metadata
- demo-mini/ – sample demo data
- CI: .github/workflows/tests.yaml
- Packaging: setup.py, requirements.txt

##### README for architecture clues
- Describes two main workflows:
1. Data generation: uses assets + config to build cladder‐v1
2. Model evaluation: runs LLMs on questions and post‐processes answers
- Mentions template files, output formats, and evaluation metrics

##### Main components
a. Inputs
- Assets (assets/stories/*.yml)
- Config files (config/*.yaml)
- Optional external dataset (HuggingFace URL or local zip)
b. Data Generation
- Generator module (causalbenchmark/generator.py)
- Builds causal graphs (causalbenchmark/graphs/)
- Samples questions via templates (causalbenchmark/verbal/)
- Produces JSON files in data/ or demo-mini/
c. Evaluation Pipeline
- Eval scripts (causalbenchmark/eval/run_*.py)
- LLM APIs (OpenAI, LLaMA wrappers)
- Prompt assembly: background + given_info + question
- Raw outputs → Postprocessing (run_llama_postprocess.py)
d. Analysis & Visualization
- Formatting module (causalbenchmark/formatting.py)
- Stats & plotting (causalbenchmark/eval/data_stats_plot.py, graphs/)
- Output: tables, confusion matrices, metrics
e. Tests & CI
- Unit tests (causalbenchmark/test/)
- GitHub Actions CI

##### Relationships and data flow
- Assets + Config → Generator → Dataset (JSON)
- Dataset → Eval scripts → LLM endpoints → Raw model outputs
- Raw outputs → Postprocessing → Formatted results
- Formatted results → Stats & Graphs → Final evaluation artifacts
- Tests exercise all code modules; CI enforces quality

##### Architectural patterns and design principles
- Modular package structure by responsibility (generator, eval, graphs, queries)
- Configuration‐driven workflows (YAML files control behavior)
- Pluggable LLM backends (separate run_*.py for different models)
- Clear separation: data generation vs. evaluation vs. analysis
- Test coverage for each submodule

##### Technology highlights
- Python 3.x, setuptools packaging
- YAML parsing for configs & assets
- HuggingFace dataset support (optional)
- LLM API clients (OpenAI, llama_cpp, etc.)
- Matplotlib/Seaborn or Plotly for visualization

##### Implementation detail callouts
- Role of causalbenchmark/queries for implementing each question type
- Plotting modules in causalbenchmark/graphs for generating study figures
- Mention test harness under causalbenchmark/test to maintain code reliability

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Inputs
    Assets["Assets (Story Templates)"]:::config
    click Assets "https://github.com/causalnlp/cladder/tree/main/assets/stories/"
    Configs["Configuration Files"]:::config
    click Configs "https://github.com/causalnlp/cladder/tree/main/config/"
    ExternalData["External Dataset (HuggingFace/local)"]:::external

    %% Data Generation Pipeline
    subgraph "Data Generation"
        direction TB
        Generator["Generator Module"]:::internal
        click Generator "https://github.com/causalnlp/cladder/blob/main/causalbenchmark/generator.py"
        GraphBuilders["Graph Builders & Plotting"]:::internal
        click GraphBuilders "https://github.com/causalnlp/cladder/tree/main/causalbenchmark/graphs/"
        Verbal["Story Verbalizations"]:::internal
        click Verbal "https://github.com/causalnlp/cladder/tree/main/causalbenchmark/verbal/"
        DatasetJSON["cladder-v1-questions.json"]:::data
        click DatasetJSON "https://github.com/causalnlp/cladder/blob/main/data/cladder-v1-questions.json"
        MetaJSON["cladder-v1-meta-models.json"]:::data
        click MetaJSON "https://github.com/causalnlp/cladder/blob/main/data/cladder-v1-meta-models.json"
        DemoData["Demo Sample Data"]:::data
        click DemoData "https://github.com/causalnlp/cladder/tree/main/demo-mini/"
    end

    %% Evaluation Pipeline
    subgraph "Model Evaluation"
        direction TB
        EvalScripts["Evaluation Scripts"]:::internal
        click EvalScripts "https://github.com/causalnlp/cladder/tree/main/causalbenchmark/eval/"
        LLMAPIs["LLM APIs (OpenAI, LLaMA)"]:::external
        RawOutputs["Raw Model Outputs"]:::data
        click RawOutputs "https://github.com/causalnlp/cladder/blob/main/data/raw_outputs.json"
    end

    %% Post-processing & Formatting
    subgraph "Post-Processing"
        direction TB
        Postprocess["Post-Processing Script"]:::internal
        click Postprocess "https://github.com/causalnlp/cladder/blob/main/causalbenchmark/eval/run_llama_postprocess.py"
        Formatting["Formatting Utilities"]:::internal
        click Formatting "https://github.com/causalnlp/cladder/blob/main/causalbenchmark/formatting.py"
        FormattedResults["Formatted Results"]:::data
    end

    %% Analysis & Visualization
    subgraph "Analysis & Visualization"
        direction TB
        StatsPlot["Data Stats & Plotting"]:::internal
        click StatsPlot "https://github.com/causalnlp/cladder/blob/main/causalbenchmark/eval/data_stats_plot.py"
        GraphPlot["Graph Plotting Modules"]:::internal
        click GraphPlot "https://github.com/causalnlp/cladder/tree/main/causalbenchmark/graphs/"
        FinalArtifacts["Metrics & Plots"]:::data
    end

    %% Tests & CI
    subgraph "Tests & CI"
        direction TB
        Tests["Unit Tests"]:::internal
        click Tests "https://github.com/causalnlp/cladder/tree/main/causalbenchmark/test/"
        CI["CI Workflow"]:::internal
        click CI "https://github.com/causalnlp/cladder/blob/main/.github/workflows/tests.yaml"
    end

    %% CLI & Packaging
    CLI["CLI Entry Point (main.py)"]:::internal
    click CLI "https://github.com/causalnlp/cladder/blob/main/main.py"
    Packaging["Packaging & Dependencies"]:::internal
    click Packaging "https://github.com/causalnlp/cladder/blob/main/setup.py"

    %% Data Flow
    Assets -->|YAML Templates| Generator
    Configs -->|YAML Configs| Generator
    ExternalData -->|Optional Dataset| Generator

    Generator -->|"Graph objects"| GraphBuilders
    GraphBuilders --> Verbal
    Verbal -->|"JSON Questions"| DatasetJSON
    Verbal --> MetaJSON
    Verbal --> DemoData

    DatasetJSON --> EvalScripts
    MetaJSON --> EvalScripts
    DemoData --> EvalScripts
    EvalScripts -->|"Prompts (JSON)"| LLMAPIs
    LLMAPIs -->|"Raw Text"| RawOutputs

    RawOutputs --> Postprocess
    Postprocess --> Formatting
    Formatting --> FormattedResults

    FormattedResults --> StatsPlot
    FormattedResults --> GraphPlot
    StatsPlot --> FinalArtifacts
    GraphPlot --> FinalArtifacts

    %% Tests & CI connections
    Tests --> Generator
    Tests --> EvalScripts
    Tests --> Formatting
    CI --> Tests

    %% CLI & Packaging usage
    CLI --> Generator
    Packaging --> Generator

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef data fill:#90EE90,stroke:#000,stroke-width:1px
    classDef config fill:#FFA500,stroke:#000,stroke-width:1px
    classDef external fill:#F0F8FF,stroke:#000,stroke-width:1px
```
