How ahmadhatahet/hil-fault-test-gen works
Project Type and Purpose
• This is a research‐oriented LLM-powered fault test‐case generation and evaluation tool for real-time fault injection in automotive software systems.
• It ingests functional safety requirements (Excel files), uses prompt templates + LLM APIs to generate fault injection test cases (TCs), runs those TCs on a hardware‐in‐the‐loop (HIL) platform, and analyzes the outcomes.
Main Components
a. Data Layer
• Directory: data/
• Contents: Excel spreadsheets of actuator/sensor requirements and examples
• Role: Source of functional safety requirements (FSRs) and ground‐truth examples
b. Configuration Layer
• Files: .env.template → .env, pyproject.toml, uv.lock
• Role: Defines environment variables (API keys, model endpoints), Python dependencies, and the project manager (“uv”) setup
c. Core Library (“TestGen”)
• Package: testgen/
• Modules:
– prompts/ (prompt templates for sensors, actuators, system/user prompts, bulk vs. single)
– analysis_utils.py, utils.py (data loading, formatting, metrics)
• Role: Encapsulates LLM interaction logic, prompt management, and output parsing
d. Orchestration Layer (Notebooks & Scripts)
• Files: *.ipynb (e.g., experiment_bulk.ipynb, experiment_all_single.ipynb, analyze_actuators.ipynb, results_analysis.ipynb)
• Role: Runs experiments in various modes (single, bulk, dropped, actuator+sensor), invokes TestGen functions, collects results, visualizes data
e. External Services
• LLM Providers:
– OpenAI (gpt-4o, gpt-4o-mini), Ollama (phi4), local Llama models, Qwen2.5
– Configured via .env
• HIL System
– A high-fidelity automotive simulator / hardware-in-the-loop rig (external to repo)
• Role: Execute generated fault injection test cases in real time
f. Output & Reporting Layer
• Directories:
– results/ (JSON files of TC execution outcomes and accuracy metrics)
– tables/ (LaTeX tables summarizing stats)
– figures/ (plots generated by notebooks)
• Role: Persist raw and aggregated results for analysis and publication
Relationships and Data Flows
1. Data Ingestion
[Data Layer] → loads Excel FSRs/examples → feeds
2. Test Case Generation
[Orchestration (Notebook)] calls → [TestGen Library]
• Passes in FSRs + prompt template
• TestGen calls → [LLM API Service]
• Receives generated fault TCs
3. Real-Time Execution
Generated TCs → sent to → [External HIL System]
HIL returns execution logs/results → saved in → results/
4. Analysis & Reporting
[Analysis Notebooks] load results/ → generate metrics → output to tables/ and figures/
Architectural Patterns & Principles
• Pipeline Pattern: linear stages (ingest → generate → execute → analyze)
• Modular Separation: core library (testgen) decoupled from orchestration (.ipynb)
• Pluggable Backends: multiple LLM providers via injectable configuration (.env)
• Reproducibility: environment and dependencies locked via uv, pyproject.toml, uv.lock