How iehok/dualcse works
1. What this repo is
DualCSE is a research codebase for learning and evaluating dual semantic representations: one geared toward explicit meaning and one toward implicit meaning. The repository is organized around a full experiment pipeline: dataset ingestion, preprocessing, model training, offline evaluation, and external LLM baselines.
2. Primary execution boundary
There are two main experiment paths:
- Local DualCSE training/evaluation under src/
- LLM baseline + scoring under src/llm_* and src/test_accuracy_llm.py
This is an important architectural boundary because the first path trains a model in-repo, while the second depends on external API providers.
3. Data sources and external datasets
The project consumes two external datasets:
- INLI cloned into datasets/inli
- ImpScore/Wang’s dataset downloaded as datasets/impscore/all_data.csv
These are not generated by the repo; the repo expects them to be placed in fixed locations and then standardized before training or evaluation.
4. Preprocessing subsystem
src/prepare_data.py is the data normalization step. Its job is to convert the raw dataset formats into a common structure that the rest of the code can use. This preprocessing step is a hard dependency for training and testing, so it sits between raw dataset acquisition and all downstream experiments.
5. Dataset and loading layer
src/datasets.py and src/data_loader.py form the data-access boundary. Together they likely define task-specific dataset objects and batching/iteration logic for training and evaluation. src/config.py probably centralizes dataset/model/run settings so the loaders and trainers can stay aligned.
6. Modeling layer
src/models.py contains the core DualCSE architecture and related model components. Based on the paper title and repo structure, this is where the explicit/implicit embedding objectives and contrastive learning machinery live. This module is the central computational dependency for training and most in-repo evaluations.
7. Training pipeline
src/train.py is the main training entry point, invoked by scripts/train.sh. The training flow appears to be: load standardized data → build dataset/dataloader → instantiate model → optimize embeddings/contrastive objectives → save a run identified by run_name. src/utils.py likely supports logging, checkpointing, metric computation, and reproducibility.
8. Evaluation for the learned model
There are dedicated evaluation scripts for the trained DualCSE model:
- src/test_rte.py for RTE-style evaluation
- src/test_implicitness_scoring.py for implicitness scoring
- src/test_implicitness_scoring_length.py for length-related analysis
- src/test_implicitness_scoring_pretrained.py for comparing against pretrained baselines
- src/test_rte_simcse.py for a SimCSE comparison point
These form the post-training validation layer and are separated by task, baseline, or analysis variant.
9. LLM baseline subsystem
src/llm_baseline.py, src/llm_data_prep.py, src/llms_api.py, and src/test_accuracy_llm.py implement the external-LLM experiment path. The flow is: prepare prompts/data → call a selected model provider via API → store responses in spreadsheet-like output → compute accuracy from those results. This subsystem is isolated from the trained DualCSE model and serves as a comparison baseline.
10. Prompting and provider integration
prompt/prompt.txt stores the prompt template used for LLM experiments. src/llms_api.py is the integration boundary to multiple providers, with API keys supplied via .env as documented in the README. Supported providers include OpenAI, DeepSeek, Gemini, Claude, and Mistral, which makes the repo multi-vendor at the experiment layer.
11. Analysis and reporting
src/analyze_data.py and src/analyze_retrieval_results.py are downstream analysis tools. They likely summarize dataset properties, retrieval behavior, or experiment outputs rather than driving core training. These scripts are useful for post-hoc interpretation and paper figures/results, but they are not part of the main training graph.
12. Operational tooling
scripts/train.sh is the main shell wrapper for reproducible training runs. requirements.txt defines the Python environment, while README.md provides the required filesystem layout and setup steps. The figures/ directory stores paper assets rather than runtime artifacts.
13. How the pieces fit together
The end-to-end flow is:
raw datasets in datasets/ → prepare_data standardization → loaders/datasets → DualCSE model training → task-specific evaluation scripts.
In parallel, the LLM path is:
prompt template + prepared data → API calls → saved predictions/results → accuracy/evaluation scripts.
14. Key architecture takeaway
The repo is best viewed as a research workflow with three layers: data preparation, modeling/evaluation, and LLM baseline comparison. The strongest boundary is between the in-repo learned embedding model and the external API-driven baseline system; both consume the same task framing but produce results through very different runtimes.
Main entry points: train.py Train · prepare_data.py Prepare data · llm_baseline.py Baseline run · config.py Config
Components 24
Operational
Project setup, docs, and run wrappers.
- README · Setup, dataset layout, and experiment instructions.
README.md - Dependencies · Python package requirements for the repo.
requirements.txt - Train script · Wrapper for reproducible training runs.
scripts/train.sh
Data Pipeline
Raw datasets, standardization, and loading for experiments.
- INLI data · External dataset expected under datasets/inli.
- ImpScore data · External CSV expected at datasets/impscore/all_data.csv.
- Prepare data · Normalizes raw dataset formats into a shared structure.
src/prepare_data.py - Datasets · Task-specific dataset objects and examples.
src/datasets.py - Loader · Batching and iteration for training and evaluation.
src/data_loader.py - Config · Central run, dataset, and model configuration.
src/config.py
DualCSE
In-repo learned representation training and evaluation.
- DualCSE · Core explicit/implicit representation model and objectives.
src/models.py - Train · Main training entry point for the learned model.
src/train.py - RTE eval · Task evaluation for the trained model.
src/test_rte.py - Implicitness · Implicitness scoring evaluation.
src/test_implicitness_scoring.py - Length analysis · Implicitness scoring by length.
src/test_implicitness_scoring_length.py - Pretrained compare · Comparison against pretrained baselines.
src/test_implicitness_scoring_pretrained.py - SimCSE eval · RTE comparison against SimCSE.
src/test_rte_simcse.py - Utils · Shared support for logging, checkpoints, and metrics.
src/utils.py
LLM Baseline
External-model comparison pipeline driven by prompts and APIs.
- Prompt · Prompt template used for LLM experiments.
prompt/prompt.txt - LLM prep · Prepares data and prompts for external model runs.
src/llm_data_prep.py - LLM API · Integration boundary to external model providers.
src/llms_api.py - Baseline run · Runs the external-LLM comparison pipeline.
src/llm_baseline.py - LLM accuracy · Computes accuracy from saved LLM outputs.
src/test_accuracy_llm.py
Analysis
Post-hoc inspection and result summaries.
- Data analysis · Summarizes dataset properties and structure.
src/analyze_data.py - Retrieval analysis · Summarizes retrieval results and experiment outputs.
src/analyze_retrieval_results.py
Connections 24
- README → Prepare data: describes · Documents dataset layout and preprocessing steps.
- README → Train script: documents · Explains the training wrapper and run flow.
- Dependencies → Train: supports · Defines the runtime environment used by scripts.
- INLI data → Prepare data: input · Raw INLI dataset is standardized for downstream use.
- ImpScore data → Prepare data: input · Raw ImpScore CSV is standardized for downstream use.
- Prepare data → Datasets: feeds · Produces normalized examples consumed by dataset objects.
- Datasets → Loader: used by · Dataset objects are batched and iterated for experiments.
- Config → Datasets: configures · Controls dataset and run selection.
- Config → Train: configures · Provides training hyperparameters and run settings.
- Config → Baseline run: configures · Controls baseline run settings and provider choice.
- Loader → Train: supplies batches · Provides training and evaluation batches.
- DualCSE → Train: core dependency · Training instantiates and optimizes the DualCSE model.
- Utils → Train: supports · Handles common training utilities.
- Train → RTE eval: evaluated by · Trained model is validated on RTE-style tasks.
- Train → Implicitness: evaluated by · Trained model is validated on implicitness scoring.
- Train → Length analysis: analyzed by · Training outputs are analyzed by sentence length.
- Train → Pretrained compare: compared by · Trained model is compared against pretrained baselines.
- Train → SimCSE eval: compared by · Trained model is compared against SimCSE.
- Prompt → LLM prep: drives · Prompt template shapes the prepared LLM inputs.
- LLM prep → LLM API: calls · Prepared prompts are sent to provider APIs.
- LLM API → Baseline run: provides responses · External API completions are consumed by the baseline runner.
- Baseline run → LLM accuracy: scored by · Saved predictions are turned into accuracy metrics.
- Prepare data → Data analysis: inspected by · Prepared data can be summarized for analysis.
- Baseline run → Retrieval analysis: inspected by · Baseline outputs can be summarized post hoc.