How sdkdeepa/ai-quality-gate works
AI Quality Gate is a release-readiness platform, not a chatbot: engineers submit golden-dataset evaluation runs, inspect normalized results, then apply release policy and optional baseline comparison to produce an auditable PASS/WARN/BLOCK decision. The graph includes the separate RAG system-under-test, framework/provider signal sources, persistence, reports, and the read-oriented dashboard. Documented framework and tracing capabilities are included; links without direct source evidence are omitted. Unsampled API and integration internals are represented only where README or indexed references support them.
Main entry points: policy_service.py Policy service · main.py FastAPI app · runner.py Evaluation runner · evaluation_service.py Evaluation service
Components 29
Engineering dashboard
- Dashboard pages
frontend/src/pages - API client
frontend/src/api/client.ts
API and orchestration
- FastAPI app
backend/app/main.py - Evaluation endpoints
backend/app/api/evaluations.py - Dataset endpoints
backend/app/api/datasets.py - Gate endpoints
backend/app/api/gate.py - Report endpoints
backend/app/api/reports.py - Dataset service
backend/app/services/dataset_service.py - Evaluation service
backend/app/services/evaluation_service.py
Evaluation signals
- Evaluation runner
backend/app/evaluation/runner.py - Deterministic evaluators
backend/app/evaluation/deterministic.py - Framework evaluators
backend/app/evaluation - Model providers
backend/app/providers - Case results
backend/app/domain/case_result.py
Release decisions
- Policy service
backend/app/services/policy_service.py - Policy engine
backend/app/policy/engine.py - Release policies
backend/app/domain/release_policy.py - Baselines
backend/app/domain/baseline.py - Gate decision
backend/app/domain/gate_decision.py - Report generation
backend/app/services/report_service.py
RAG system under test
- RAG service
backend/app/services/rag_service.py - RAG pipeline
backend/app/rag/pipeline.py - Retriever
backend/app/rag/retriever.py - Corpus ingestion
backend/app/rag/corpus_service.py - Vector store
backend/app/rag/vector_store.py
Data and observability
- Run and case storage
backend/app/repositories/in_memory.py - Policy and decision storage
backend/app/repositories/sqlite.py - Tracing
backend/app/observability/tracing.py
Other components
- Engineer
Connections 34
- Engineer → Dashboard pages: uses
- Dashboard pages → API client: requests data
frontend/src/pages/RunDetail.tsx - API client → FastAPI app: calls API
frontend/src/api/client.ts - FastAPI app → Evaluation endpoints: registers routes
backend/app/main.py - FastAPI app → Dataset endpoints: registers routes
backend/app/main.py - FastAPI app → Gate endpoints: registers routes
backend/app/main.py - FastAPI app → Report endpoints: registers routes
backend/app/main.py - Evaluation endpoints → Evaluation service: dispatches
- Dataset endpoints → Dataset service: dispatches
- Evaluation service → Dataset service: loads datasets
- Evaluation service → Evaluation runner: runs evaluation
- Evaluation runner → Model providers: requests responses
backend/app/evaluation/runner.py - Evaluation runner → Deterministic evaluators: executes metrics
backend/app/main.py - Evaluation runner → Framework evaluators: executes optional metrics
backend/app/main.py - Evaluation runner → Case results: produces results
backend/app/evaluation/runner.py - Evaluation service → Run and case storage: stores runs
- Evaluation service → Case results: stores results
- Gate endpoints → Policy service: dispatches
- Policy service → Policy engine: requests decision
backend/app/services/policy_service.py - Policy service → Run and case storage: reads run results
backend/app/services/policy_service.py - Policy service → Release policies: resolves policy
backend/app/services/policy_service.py - Policy service → Baselines: loads baseline
backend/app/services/policy_service.py - Policy service → Policy and decision storage: persists decision
backend/app/services/policy_service.py - Policy engine → Case results: evaluates metrics
backend/app/policy/engine.py - Policy engine → Gate decision: computes outcome
backend/app/policy/engine.py - Report endpoints → Report generation: dispatches
- Report generation → Policy service: loads decision data
backend/app/main.py - RAG service → RAG pipeline: answers queries
backend/app/services/rag_service.py - RAG pipeline → Retriever: retrieves context
backend/app/rag/pipeline.py - RAG pipeline → Model providers: generates answer
backend/app/rag/pipeline.py - RAG service → Evaluation runner: grades case
backend/app/services/rag_service.py - Corpus ingestion → Vector store: writes chunks
backend/app/rag/corpus_service.py - Retriever → Vector store: searches chunks
backend/app/rag/retriever.py - FastAPI app → Tracing: configures tracing
backend/app/main.py