How autoiac-project/iac-eval works
1. This repo is an offline evaluation framework for LLMs that generate Terraform Infrastructure-as-Code, centered on AWS scenarios. The core boundary is: benchmark data + templates in, model-generated IaC out, then automated scoring/judging.
2. The main artifact is the IaC-Eval dataset itself, referenced from Hugging Face and described as 458 curated questions. The repository does not embed the full dataset; instead it provides the evaluation harness and the template assets used to assess model outputs.
3. templates/ is the scenario library. Each AWS service directory contains Terraform reference material such as main.tf, tfplan.json, and often template.rego. These look like the per-task ground truth / policy checking assets used to define what a correct solution should look like.
4. evaluation/eval.py is the primary orchestration entry point for the benchmark pipeline. evaluation/config.json and the prompt templates control how a run is parameterized, which model is called, and what prompting strategy is used.
5. evaluation/models.py is the model-provider abstraction layer. The README indicates support for OpenAI, Google Gemini, Replicate-hosted code models, and a manually deployed SageMaker endpoint for MagiCoder, so the architecture is designed around pluggable LLM backends rather than one fixed inference service.
6. evaluation/prompt-templates/ and evaluation/prompt_templates.py define the instruction layer passed to models. The presence of system, few-shot, CoT, and multi-turn prompts suggests the pipeline can switch among multiple generation modes without changing the rest of the evaluation stack.
7. evaluation/metrics.py and evaluation/llm-judge-eval.py split automated assessment into two layers: deterministic metrics and LLM-based judging. This is important architecturally because the repo supports both rule-based scoring and semantic quality comparison.
8. evaluation/data.py is the data-loading and sample-shaping boundary between the dataset and the rest of the pipeline. It likely normalizes benchmark records into the structures consumed by prompt generation, model invocation, and scoring.
9. retriever/ is an optional retrieval-augmented generation subsystem. retriever/llama_index_retriever.py and its setup script indicate a separate retrieval index/database that can be preconfigured and then queried by the evaluation pipeline when RAG is enabled.
10. The repo depends on external execution and validation tooling: Terraform, OPA, AWS CLI credentials, and Python Conda packaging via environment.yml and setup.sh. This means the evaluation loop is not purely textual; it is tied to infrastructure tooling and policy evaluation.
11. evaluation/misc/ contains experiment-specific scripts rather than core runtime code. The subfolders for ablations, multiple-sample runs, complete-dataset measurement, and SageMaker setup show how the authors reproduced paper experiments and deployed special inference infrastructure.
12. evaluation/logs/ and licenses/ are supporting boundaries for run artifacts and compliance metadata. They do not drive the benchmark logic, but they matter for reproducibility, auditability, and packaging.
13. Overall data flow: benchmark/task definition from dataset and templates -> optional retrieval -> prompt assembly -> LLM generation -> Terraform/OPA-based validation and/or LLM judging -> metrics and logs. The architecture is modular enough to swap models, prompting strategies, and evaluation modes while keeping the task templates stable.
Main entry points: eval.py Eval runner · templates Scenario library · prompt_templates.py Prompt layer · config.json Run config
Components 21
Evaluation core
Benchmark orchestration, prompting, model access, and scoring.
- Eval runner · Primary benchmark entry point that coordinates loading, prompting, generation, and scoring.
evaluation/eval.py - Data loader · Loads and normalizes benchmark records for the evaluation pipeline.
evaluation/data.py - Model backends · Pluggable providers for OpenAI, Gemini, Replicate, and SageMaker-hosted models.
evaluation/models.py - Prompt layer · Prompt templates and prompt-construction logic for system, few-shot, CoT, and multi-turn modes.
evaluation/prompt_templates.py - Prompt texts · Reusable instruction templates consumed by the prompt assembly layer.
evaluation/prompt-templates - Deterministic scores · Non-LLM scoring and comparison logic for automated evaluation.
evaluation/metrics.py - LLM judge · Secondary evaluation path that uses an LLM to judge generated IaC outputs.
evaluation/llm-judge-eval.py - Run config · Parameters that control model choice, prompting strategy, and run behavior.
evaluation/config.json
Task assets
Scenario templates and reference artifacts used to define benchmark tasks.
- Scenario library · AWS scenario templates and reference artifacts for benchmark tasks.
templates - VPC tasks · VPC scenarios with multiple variants and reference plans.
templates/aws_vpc - IAM tasks · IAM policy, role, and user scenarios used for evaluation.
templates/aws_iam_policy - Storage tasks · S3 and related storage scenarios.
templates/aws_s3_bucket - Networking tasks · Subnet, load balancer, and related network scenarios.
templates/aws_subnet - Database tasks · Neptune and Redshift scenarios used in the benchmark suite.
templates/aws_neptune_cluster - Comms tasks · AWS Chime voice connector scenario templates.
templates/aws_chime_voice_connector
Optional subsystems
Retrieval augmentation and experiment-specific scripts.
- Retriever · Optional retrieval layer used to augment prompts with external context.
retriever - Experiments · Ablations, multi-sample runs, complete-dataset measurement, and SageMaker deployment helpers.
evaluation/misc - Run logs · Stored run outputs and related logging materials.
evaluation/logs
Support files
Packaging, setup, and compliance materials.
- Setup scripts · Repository-level and evaluation-specific bootstrap scripts.
setup.sh - Conda env · Pinned Python environment for running the evaluation stack.
environment.yml - Licenses · License and attribution materials.
licenses
Connections 18
- Run config → Eval runner: parameters · Controls run mode, model selection, and prompting behavior.
- Eval runner → Data loader: loads · Fetches and normalizes benchmark cases for evaluation.
- Eval runner → Prompt layer: builds prompts · Uses prompt-construction logic to prepare model input.
- Prompt layer → Prompt texts: reads templates · Consumes system, few-shot, CoT, and multi-turn prompt text assets.
- Eval runner → Model backends: calls model · Invokes the selected LLM backend for generation.
- Eval runner → Retriever: optional RAG · Queries retrieval subsystem when enabled.
- Eval runner → Deterministic scores: scores · Computes deterministic evaluation metrics on outputs.
- Eval runner → LLM judge: judges · Routes outputs to LLM-based judging when enabled.
- Scenario library → VPC tasks: contains · VPC task variants and reference assets.
- Scenario library → IAM tasks: contains · IAM task variants and reference assets.
- Scenario library → Storage tasks: contains · Storage task variants and reference assets.
- Scenario library → Networking tasks: contains · Networking task variants and reference assets.
- Scenario library → Database tasks: contains · Database task variants and reference assets.
- Scenario library → Comms tasks: contains · Communication scenario assets.
- Setup scripts → Conda env: boots env · Environment and setup scripts prepare the local runtime.
- Experiments → Eval runner: reuses · Experiment scripts reuse the main evaluation pipeline.
- Run logs → Eval runner: records · Captures evaluation outputs and run artifacts.
- Licenses → Scenario library: documents · Packaging and compliance boundary for benchmark assets.