# autoiac-project/iac-eval architecture

> Architecture diagram of the GitHub repository autoiac-project/iac-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/autoiac-project/iac-eval
- Repository: https://github.com/autoiac-project/iac-eval

## Overview

1. This repo is an offline evaluation framework for LLMs that generate Terraform Infrastructure-as-Code, centered on AWS scenarios. The core boundary is: benchmark data + templates in, model-generated IaC out, then automated scoring/judging.

2. The main artifact is the IaC-Eval dataset itself, referenced from Hugging Face and described as 458 curated questions. The repository does not embed the full dataset; instead it provides the evaluation harness and the template assets used to assess model outputs.

3. `templates/` is the scenario library. Each AWS service directory contains Terraform reference material such as `main.tf`, `tfplan.json`, and often `template.rego`. These look like the per-task ground truth / policy checking assets used to define what a correct solution should look like.

4. `evaluation/eval.py` is the primary orchestration entry point for the benchmark pipeline. `evaluation/config.json` and the prompt templates control how a run is parameterized, which model is called, and what prompting strategy is used.

5. `evaluation/models.py` is the model-provider abstraction layer. The README indicates support for OpenAI, Google Gemini, Replicate-hosted code models, and a manually deployed SageMaker endpoint for MagiCoder, so the architecture is designed around pluggable LLM backends rather than one fixed inference service.

6. `evaluation/prompt-templates/` and `evaluation/prompt_templates.py` define the instruction layer passed to models. The presence of system, few-shot, CoT, and multi-turn prompts suggests the pipeline can switch among multiple generation modes without changing the rest of the evaluation stack.

7. `evaluation/metrics.py` and `evaluation/llm-judge-eval.py` split automated assessment into two layers: deterministic metrics and LLM-based judging. This is important architecturally because the repo supports both rule-based scoring and semantic quality comparison.

8. `evaluation/data.py` is the data-loading and sample-shaping boundary between the dataset and the rest of the pipeline. It likely normalizes benchmark records into the structures consumed by prompt generation, model invocation, and scoring.

9. `retriever/` is an optional retrieval-augmented generation subsystem. `retriever/llama_index_retriever.py` and its setup script indicate a separate retrieval index/database that can be preconfigured and then queried by the evaluation pipeline when RAG is enabled.

10. The repo depends on external execution and validation tooling: Terraform, OPA, AWS CLI credentials, and Python Conda packaging via `environment.yml` and `setup.sh`. This means the evaluation loop is not purely textual; it is tied to infrastructure tooling and policy evaluation.

11. `evaluation/misc/` contains experiment-specific scripts rather than core runtime code. The subfolders for ablations, multiple-sample runs, complete-dataset measurement, and SageMaker setup show how the authors reproduced paper experiments and deployed special inference infrastructure.

12. `evaluation/logs/` and `licenses/` are supporting boundaries for run artifacts and compliance metadata. They do not drive the benchmark logic, but they matter for reproducibility, auditability, and packaging.

13. Overall data flow: benchmark/task definition from dataset and templates -> optional retrieval -> prompt assembly -> LLM generation -> Terraform/OPA-based validation and/or LLM judging -> metrics and logs. The architecture is modular enough to swap models, prompting strategies, and evaluation modes while keeping the task templates stable.

Main entry points: [`evaluation/eval.py`](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/eval.py) (Eval runner), [`templates`](https://github.com/autoiac-project/iac-eval/tree/main/templates) (Scenario library), [`evaluation/prompt_templates.py`](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/prompt_templates.py) (Prompt layer), [`evaluation/config.json`](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/config.json) (Run config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Evaluation core"]
  node_eval_entry["Eval runner<br/>orchestrator<br/>[eval.py]"]
  node_eval_data["Data loader<br/>dataset adapter<br/>[data.py]"]
  node_eval_models["Model backends<br/>llm adapters<br/>[models.py]"]
  node_prompt_defs["Prompt layer<br/>prompt assembly"]
  node_prompt_texts["Prompt texts<br/>prompt assets"]
  node_metrics["Deterministic scores<br/>rule metrics<br/>[metrics.py]"]
  node_llm_judge["LLM judge<br/>semantic judge<br/>[llm-judge-eval.py]"]
  node_config["Run config<br/>experiment config<br/>[config.json]"]
end

subgraph group_assets["Task assets"]
  node_templates_root[("Scenario library<br/>task templates")]
  node_aws_vpc["VPC tasks<br/>aws scenario"]
  node_aws_iam["IAM tasks<br/>aws scenario"]
  node_aws_storage["Storage tasks<br/>aws scenario"]
  node_aws_networking["Networking tasks<br/>aws scenario"]
  node_aws_db["Database tasks<br/>aws scenario"]
  node_aws_communication["Comms tasks<br/>aws scenario"]
end

subgraph group_optional["Optional subsystems"]
  node_retriever["Retriever<br/>rag subsystem"]
  node_misc_runs["Experiments<br/>ablation scripts"]
  node_logs["Run logs<br/>artifacts"]
end

subgraph group_support["Support files"]
  node_setup["Setup scripts<br/>environment setup<br/>[setup.sh]"]
  node_env["Conda env<br/>python env<br/>[environment.yml]"]
  node_licenses["Licenses<br/>compliance"]
end

node_config -->|"parameters"| node_eval_entry
node_eval_entry -->|"loads"| node_eval_data
node_eval_entry -->|"builds prompts"| node_prompt_defs
node_prompt_defs -->|"reads templates"| node_prompt_texts
node_eval_entry -->|"calls model"| node_eval_models
node_eval_entry -.->|"optional RAG"| node_retriever
node_eval_entry -->|"scores"| node_metrics
node_eval_entry -->|"judges"| node_llm_judge
node_templates_root -->|"contains"| node_aws_vpc
node_templates_root -->|"contains"| node_aws_iam
node_templates_root -->|"contains"| node_aws_storage
node_templates_root -->|"contains"| node_aws_networking
node_templates_root -->|"contains"| node_aws_db
node_templates_root -->|"contains"| node_aws_communication
node_setup -->|"boots env"| node_env
node_misc_runs -.->|"reuses"| node_eval_entry
node_logs -.->|"records"| node_eval_entry
node_licenses -.->|"documents"| node_templates_root

click node_eval_entry "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/eval.py"
click node_eval_data "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/data.py"
click node_eval_models "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/models.py"
click node_prompt_defs "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/prompt_templates.py"
click node_prompt_texts "https://github.com/autoiac-project/iac-eval/tree/main/evaluation/prompt-templates"
click node_metrics "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/metrics.py"
click node_llm_judge "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/llm-judge-eval.py"
click node_config "https://github.com/autoiac-project/iac-eval/blob/main/evaluation/config.json"
click node_templates_root "https://github.com/autoiac-project/iac-eval/tree/main/templates"
click node_aws_vpc "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_vpc"
click node_aws_iam "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_iam_policy"
click node_aws_storage "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_s3_bucket"
click node_aws_networking "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_subnet"
click node_aws_db "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_neptune_cluster"
click node_aws_communication "https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_chime_voice_connector"
click node_retriever "https://github.com/autoiac-project/iac-eval/tree/main/retriever"
click node_misc_runs "https://github.com/autoiac-project/iac-eval/tree/main/evaluation/misc"
click node_logs "https://github.com/autoiac-project/iac-eval/tree/main/evaluation/logs"
click node_setup "https://github.com/autoiac-project/iac-eval/blob/main/setup.sh"
click node_env "https://github.com/autoiac-project/iac-eval/blob/main/environment.yml"
click node_licenses "https://github.com/autoiac-project/iac-eval/tree/main/licenses"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_eval_entry,node_eval_data,node_eval_models,node_prompt_defs,node_prompt_texts,node_metrics,node_llm_judge,node_config toneBlue
class node_templates_root,node_aws_vpc,node_aws_iam,node_aws_storage,node_aws_networking,node_aws_db,node_aws_communication toneAmber
class node_retriever,node_misc_runs,node_logs toneMint
class node_setup,node_env,node_licenses toneRose
```

## Components

### Evaluation core

Benchmark orchestration, prompting, model access, and scoring.

- [Eval runner](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/eval.py): orchestrator · Primary benchmark entry point that coordinates loading, prompting, generation, and scoring. · `evaluation/eval.py`
- [Data loader](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/data.py): dataset adapter · Loads and normalizes benchmark records for the evaluation pipeline. · `evaluation/data.py`
- [Model backends](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/models.py): llm adapters · Pluggable providers for OpenAI, Gemini, Replicate, and SageMaker-hosted models. · `evaluation/models.py`
- [Prompt layer](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/prompt_templates.py): prompt assembly · Prompt templates and prompt-construction logic for system, few-shot, CoT, and multi-turn modes. · `evaluation/prompt_templates.py`
- [Prompt texts](https://github.com/autoiac-project/iac-eval/tree/main/evaluation/prompt-templates): prompt assets · Reusable instruction templates consumed by the prompt assembly layer. · `evaluation/prompt-templates`
- [Deterministic scores](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/metrics.py): rule metrics · Non-LLM scoring and comparison logic for automated evaluation. · `evaluation/metrics.py`
- [LLM judge](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/llm-judge-eval.py): semantic judge · Secondary evaluation path that uses an LLM to judge generated IaC outputs. · `evaluation/llm-judge-eval.py`
- [Run config](https://github.com/autoiac-project/iac-eval/blob/main/evaluation/config.json): experiment config · Parameters that control model choice, prompting strategy, and run behavior. · `evaluation/config.json`

### Task assets

Scenario templates and reference artifacts used to define benchmark tasks.

- [Scenario library](https://github.com/autoiac-project/iac-eval/tree/main/templates): task templates · AWS scenario templates and reference artifacts for benchmark tasks. · `templates`
- [VPC tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_vpc): aws scenario · VPC scenarios with multiple variants and reference plans. · `templates/aws_vpc`
- [IAM tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_iam_policy): aws scenario · IAM policy, role, and user scenarios used for evaluation. · `templates/aws_iam_policy`
- [Storage tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_s3_bucket): aws scenario · S3 and related storage scenarios. · `templates/aws_s3_bucket`
- [Networking tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_subnet): aws scenario · Subnet, load balancer, and related network scenarios. · `templates/aws_subnet`
- [Database tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_neptune_cluster): aws scenario · Neptune and Redshift scenarios used in the benchmark suite. · `templates/aws_neptune_cluster`
- [Comms tasks](https://github.com/autoiac-project/iac-eval/tree/main/templates/aws_chime_voice_connector): aws scenario · AWS Chime voice connector scenario templates. · `templates/aws_chime_voice_connector`

### Optional subsystems

Retrieval augmentation and experiment-specific scripts.

- [Retriever](https://github.com/autoiac-project/iac-eval/tree/main/retriever): rag subsystem · Optional retrieval layer used to augment prompts with external context. · `retriever`
- [Experiments](https://github.com/autoiac-project/iac-eval/tree/main/evaluation/misc): ablation scripts · Ablations, multi-sample runs, complete-dataset measurement, and SageMaker deployment helpers. · `evaluation/misc`
- [Run logs](https://github.com/autoiac-project/iac-eval/tree/main/evaluation/logs): artifacts · Stored run outputs and related logging materials. · `evaluation/logs`

### Support files

Packaging, setup, and compliance materials.

- [Setup scripts](https://github.com/autoiac-project/iac-eval/blob/main/setup.sh): environment setup · Repository-level and evaluation-specific bootstrap scripts. · `setup.sh`
- [Conda env](https://github.com/autoiac-project/iac-eval/blob/main/environment.yml): python env · Pinned Python environment for running the evaluation stack. · `environment.yml`
- [Licenses](https://github.com/autoiac-project/iac-eval/tree/main/licenses): compliance · License and attribution materials. · `licenses`

## Connections

- Run config → Eval runner: parameters · Controls run mode, model selection, and prompting behavior.
- Eval runner → Data loader: loads · Fetches and normalizes benchmark cases for evaluation.
- Eval runner → Prompt layer: builds prompts · Uses prompt-construction logic to prepare model input.
- Prompt layer → Prompt texts: reads templates · Consumes system, few-shot, CoT, and multi-turn prompt text assets.
- Eval runner → Model backends: calls model · Invokes the selected LLM backend for generation.
- Eval runner → Retriever: optional RAG · Queries retrieval subsystem when enabled.
- Eval runner → Deterministic scores: scores · Computes deterministic evaluation metrics on outputs.
- Eval runner → LLM judge: judges · Routes outputs to LLM-based judging when enabled.
- Scenario library → VPC tasks: contains · VPC task variants and reference assets.
- Scenario library → IAM tasks: contains · IAM task variants and reference assets.
- Scenario library → Storage tasks: contains · Storage task variants and reference assets.
- Scenario library → Networking tasks: contains · Networking task variants and reference assets.
- Scenario library → Database tasks: contains · Database task variants and reference assets.
- Scenario library → Comms tasks: contains · Communication scenario assets.
- Setup scripts → Conda env: boots env · Environment and setup scripts prepare the local runtime.
- Experiments → Eval runner: reuses · Experiment scripts reuse the main evaluation pipeline.
- Run logs → Eval runner: records · Captures evaluation outputs and run artifacts.
- Licenses → Scenario library: documents · Packaging and compliance boundary for benchmark assets.
