How timauzi/llm-x-cxg works
Project Type and Purpose
• This is a research‐oriented Python project (not a live service) for probing construction grammar (CxG) in large language models (LLMs).
• Two primary tasks:
a. Construction Detection (finding predefined constructions in corpora)
b. Text Generation (few‐shot or zero‐shot example generation guided by construction definitions)
High‐Level Architecture
At the highest level, the system comprises:
A. Data Layer
• Raw corpora, construction annotations, pseudoword definitions
• Experiment outputs (detection results, generated examples, similarity measures)
B. Preprocessing & Pseudoword Module
• Scripts to produce pseudoword embeddings (based on BERT tooling)
C. Experiment Pipelines
1. Detection Pipeline (LLM + CxG matcher → detection TSVs)
2. Generation Pipeline (few‐shot templates + LLM → generated TSVs)
3. Similarity Analysis (embedding comparisons → similarity TSVs)
D. Evaluation Layer
• Jupyter notebooks for automated and manual evaluation of outputs
E. Environment & Config
• Conda environment (llm-cxg.txt) for reproducibility
Main Components & Interactions
• Data Sources (data/annotations, data/detection, etc.) feed into Preprocessing & Experiment Pipelines.
• Preprocessing & Pseudoword Module (src/pseudowords/) generates augmented embeddings used by both detection and generation pipelines.
• Experiment Pipelines invoke Huggingface LLMs (BERT, Llama 2, mBART) with varying shot settings and constructions (0–5 shots).
• Pipelines write outputs back into data/detection/, data/generation/, data/similarities/.
• Evaluation Notebooks (src/notebooks/) read these outputs, compute metrics, produce figures, and support manual inspection.
Architectural Patterns & Design Principles
• Modular Pipeline Pattern: clear separation of data ingestion, model invocation, and evaluation.
• Directory‐as‐Module: each subfolder (e.g., data/generation, src/pseudowords) encapsulates its function.
• Versioned Data Artifacts: TSV files version experiments by model, attempts, and shot counts.
Technologies & Frameworks
• Python + Jupyter Notebooks
• Huggingface Transformers (BERT, Llama 2, mBART)
• Pandas/TSV for data interchange
• Conda for environment management
• Bash scripts (e.g., script_bsb.sh) for batch runs