# finansminister/llm-code-security-audit-script architecture

> Architecture diagram of the GitHub repository finansminister/llm-code-security-audit-script, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/finansminister/llm-code-security-audit-script
- Repository: https://github.com/finansminister/llm-code-security-audit-script

## Overview

1. **Purpose and top-level flow**  
   This repo is an orchestration pipeline for auditing Python code generated by LLMs. The main path is: prompt dataset → LLM generation → code extraction/cleanup → static security scan with CodeQL → CWE/OWASP categorization → aggregate statistics and audit logs.

2. **Entry points and orchestration**  
   `main.py` is the central runner. It likely coordinates batch execution, session resumption, and handoff between generation and scanning. `freeze_hashes.py` is a separate maintenance utility used to create the baseline `master_hashes.json` for integrity checks.

3. **Configuration boundary**  
   `config.py` centralizes runtime settings, API parameters, paths, and telemetry/logging configuration. This file is the shared dependency for most subsystems, which keeps environment-specific wiring out of the core logic.

4. **Input corpus**  
   `seceval-dataset/dataset.jsonl` is the prompt source. The system appears to iterate over this dataset to generate code samples for multiple models, so the dataset is the upstream input boundary for the whole experiment.

5. **LLM access layer**  
   `src/llm/clients.py` encapsulates provider-specific API clients. The repo supports several model providers, so this layer isolates authentication, endpoint differences, and model selection from the rest of the pipeline.  
   `src/llm/generator.py` drives the batch prompting workflow and persists model responses for later analysis.

6. **Core safety and normalization layer**  
   `src/core/integrity.py` handles source/state validation using hashes against `master_hashes.json`. This is a guardrail to detect drift in the codebase or experiment artifacts.  
   `src/core/parser.py` sanitizes raw LLM output into usable Python source, which forms the handoff from text generation to static analysis.

7. **Static analysis subsystem**  
   `src/scanners/codeql.py` wraps the CodeQL CLI and is responsible for building databases, running queries, and emitting SARIF or similar machine-readable scan results. This is the external security-analysis engine in the architecture.  
   `src/scanners/owasp.py` maps CodeQL-derived CWE findings into OWASP Top 10 categories, providing the semantic layer used by the reporting and statistics code.

8. **Audit and experiment logging**  
   `src/analysis/audit.py` provides persistent logging and tee-style console/file output so the run can be replayed or inspected after interruption. It likely records per-attempt metadata, errors, and scan outcomes as an audit trail.

9. **Statistical analysis layer**  
   `src/analysis/stats.py` aggregates the scan results across prompts/models and computes comparative statistics such as distribution tests or significance tests. This is the analytical boundary that turns raw vulnerability findings into experimental conclusions.

10. **State and resumption model**  
   The repo is designed around resumable batch processing. Generated code, scan artifacts, hashes, and logs are likely written to disk so a partial run can continue without repeating expensive LLM calls or re-scans.

11. **Important external dependencies**  
   The main external runtime dependencies are the LLM provider APIs and the CodeQL CLI. Python is the execution environment, while the repo’s own modules mostly coordinate I/O, validation, scan execution, and report aggregation.

12. **Architectural diagram takeaway**  
   A good diagram should show three major planes:  
   - **Experiment control**: `main.py`, `config.py`, `freeze_hashes.py`  
   - **Generation and security pipeline**: dataset → `src/llm` → `src/core` → `src/scanners`  
   - **Observability and analysis**: `src/analysis` + persisted artifacts (`master_hashes.json`, logs, scan outputs, stats outputs)

Main entry points: [`main.py`](https://github.com/finansminister/llm-code-security-audit-script/blob/main/main.py) (main.py), [`src/llm/generator.py`](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/llm/generator.py) (Generator), [`src/scanners/codeql.py`](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/scanners/codeql.py) (CodeQL), [`src/analysis/stats.py`](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/analysis/stats.py) (Statistics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Experiment control"]
  node_main_py["main.py<br/>orchestrator<br/>[main.py]"]
  node_config_py["config.py<br/>settings<br/>[config.py]"]
  node_freeze_hashes_py["freeze_hashes.py<br/>maintenance<br/>[freeze_hashes.py]"]
end

subgraph group_pipeline["Generation and scanning"]
  node_dataset_jsonl["Prompt set<br/>corpus<br/>[dataset.jsonl]"]
  node_llm_clients_py["LLM clients<br/>api adapters<br/>[clients.py]"]
  node_llm_generator_py["Generator<br/>batch prompting<br/>[generator.py]"]
  node_parser_py["Parser<br/>cleanup<br/>[parser.py]"]
  node_integrity_py{{"Integrity<br/>hash guard<br/>[integrity.py]"}}
  node_codeql_py["CodeQL<br/>scanner wrapper<br/>[codeql.py]"]
  node_owasp_py["OWASP map<br/>categorizer<br/>[owasp.py]"]
  node_codeql_cli(("CodeQL CLI<br/>external tool"))
end

subgraph group_analysis["Audit and analysis"]
  node_audit_py["Audit log<br/>run logging<br/>[audit.py]"]
  node_stats_py["Statistics<br/>aggregation<br/>[stats.py]"]
  node_master_hashes_json["Hash manifest<br/>artifact<br/>[master_hashes.json]"]
  node_scan_artifacts[("Scan outputs<br/>artifact store")]
  node_logs[("Run logs<br/>artifact store")]
end

node_main_py -->|"loads settings"| node_config_py
node_main_py -->|"iterates prompts"| node_dataset_jsonl
node_main_py -->|"starts generation"| node_llm_generator_py
node_main_py -->|"checks state"| node_integrity_py
node_main_py -->|"records run"| node_audit_py
node_freeze_hashes_py -->|"writes baseline"| node_master_hashes_json
node_llm_generator_py -->|"calls models"| node_llm_clients_py
node_llm_generator_py -->|"cleans output"| node_parser_py
node_llm_generator_py -->|"logs attempts"| node_audit_py
node_parser_py -->|"hands off code"| node_codeql_py
node_integrity_py -->|"verifies against"| node_master_hashes_json
node_codeql_py -->|"invokes"| node_codeql_cli
node_codeql_py -->|"emits results"| node_scan_artifacts
node_codeql_py -->|"provides findings"| node_owasp_py
node_owasp_py -->|"categorizes input"| node_stats_py
node_audit_py -->|"persists trail"| node_logs
node_stats_py -->|"reads outputs"| node_scan_artifacts
node_stats_py -->|"summarizes run"| node_logs

click node_main_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/main.py"
click node_config_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/config.py"
click node_freeze_hashes_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/freeze_hashes.py"
click node_dataset_jsonl "https://github.com/finansminister/llm-code-security-audit-script/blob/main/seceval-dataset/dataset.jsonl"
click node_llm_clients_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/llm/clients.py"
click node_llm_generator_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/llm/generator.py"
click node_parser_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/core/parser.py"
click node_integrity_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/core/integrity.py"
click node_codeql_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/scanners/codeql.py"
click node_owasp_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/scanners/owasp.py"
click node_audit_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/analysis/audit.py"
click node_stats_py "https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/analysis/stats.py"
click node_master_hashes_json "https://github.com/finansminister/llm-code-security-audit-script/blob/main/master_hashes.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_config_py,node_freeze_hashes_py toneBlue
class node_dataset_jsonl,node_llm_clients_py,node_llm_generator_py,node_parser_py,node_integrity_py,node_codeql_py,node_owasp_py,node_codeql_cli toneAmber
class node_audit_py,node_stats_py,node_master_hashes_json,node_scan_artifacts,node_logs toneMint
```

## Components

### Experiment control

Entry points, runtime settings, and maintenance utilities.

- [main.py](https://github.com/finansminister/llm-code-security-audit-script/blob/main/main.py): orchestrator · Central runner that coordinates batches, resumption, and pipeline handoff. · `main.py`
- [config.py](https://github.com/finansminister/llm-code-security-audit-script/blob/main/config.py): settings · Shared runtime configuration for paths, model parameters, and logging. · `config.py`
- [freeze_hashes.py](https://github.com/finansminister/llm-code-security-audit-script/blob/main/freeze_hashes.py): maintenance · Utility for generating the baseline hash manifest. · `freeze_hashes.py`

### Generation and scanning

Dataset-driven LLM generation, cleanup, integrity checks, and security scanning.

- [Prompt set](https://github.com/finansminister/llm-code-security-audit-script/blob/main/seceval-dataset/dataset.jsonl): corpus · Upstream prompt dataset used to drive code generation experiments. · `seceval-dataset/dataset.jsonl`
- [LLM clients](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/llm/clients.py): api adapters · Provider-specific client wrappers for model access and authentication. · `src/llm/clients.py`
- [Generator](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/llm/generator.py): batch prompting · Runs prompt batches, captures model output, and persists responses. · `src/llm/generator.py`
- [Parser](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/core/parser.py): cleanup · Sanitizes raw model output into usable Python source. · `src/core/parser.py`
- [Integrity](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/core/integrity.py): hash guard · Validates repository state and artifacts against master hashes. · `src/core/integrity.py`
- [CodeQL](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/scanners/codeql.py): scanner wrapper · Builds databases and runs security queries through the CodeQL CLI. · `src/scanners/codeql.py`
- [OWASP map](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/scanners/owasp.py): categorizer · Maps CWE-style findings into OWASP Top 10 categories. · `src/scanners/owasp.py`
- **CodeQL CLI**: external tool · External security-analysis engine invoked by the scanner wrapper.

### Audit and analysis

Logging, categorization, and aggregate statistics over scan outcomes.

- [Audit log](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/analysis/audit.py): run logging · Persistent audit trail and tee-style console/file logging for runs. · `src/analysis/audit.py`
- [Statistics](https://github.com/finansminister/llm-code-security-audit-script/blob/main/src/analysis/stats.py): aggregation · Aggregates findings and computes comparative experiment statistics. · `src/analysis/stats.py`
- [Hash manifest](https://github.com/finansminister/llm-code-security-audit-script/blob/main/master_hashes.json): artifact · Baseline hashes used for integrity validation. · `master_hashes.json`
- **Scan outputs**: artifact store · Persisted scan results and intermediate artifacts produced during analysis.
- **Run logs**: artifact store · Console/file logs and audit records from resumable runs.

## Connections

- main.py → config.py: loads settings · Main execution reads shared runtime configuration.
- main.py → Prompt set: iterates prompts · Batch workflow consumes the prompt corpus.
- main.py → Generator: starts generation · Coordinates generation runs over the dataset.
- main.py → Integrity: checks state · Uses hash-based validation to protect resumable execution.
- main.py → Audit log: records run · Writes execution progress and outcomes to the audit trail.
- freeze_hashes.py → Hash manifest: writes baseline · Creates or refreshes the integrity manifest.
- Generator → LLM clients: calls models · Uses provider adapters to request completions.
- Generator → Parser: cleans output · Normalizes raw model text into source code.
- Generator → Audit log: logs attempts · Persists per-run generation metadata and errors.
- Parser → CodeQL: hands off code · Produces source that becomes input for static analysis.
- Integrity → Hash manifest: verifies against · Compares current state with the frozen hash manifest.
- CodeQL → CodeQL CLI: invokes · Wraps the external CodeQL runtime and queries.
- CodeQL → Scan outputs: emits results · Stores scan databases, findings, and machine-readable outputs.
- CodeQL → OWASP map: provides findings · Feeds vulnerability results into semantic categorization.
- OWASP map → Statistics: categorizes input · Transforms findings into analysis-ready categories.
- Audit log → Run logs: persists trail · Writes replayable logs and audit records.
- Statistics → Scan outputs: reads outputs · Aggregates scan artifacts for comparative analysis.
- Statistics → Run logs: summarizes run · Produces analysis summaries alongside logs.
