How nethermindeth/agentarena-arbiter works
1. What this repo is
ArbiterAgent is a Python triage service for vulnerability reports. Its job is to take multiple agent-submitted findings, normalize them, deduplicate overlaps, score severity, and persist or return the consolidated result.
2. Top-level structure
The architecture is centered in Demo/app/:
- main.py is the application entrypoint and likely exposes the API surface.
- core/ contains the triage logic.
- models/ defines the input/output data shapes.
- database/ contains persistence access, specifically MongoDB handling.
- config.py, types.py, and task_utils.py hold shared runtime configuration and orchestration utilities.
3. Request/response boundary
The system’s main external boundary is a JSON-based API. Incoming finding payloads are parsed into structured models from models/finding_input.py, processed by the core pipeline, and emitted back as consolidated findings, likely using models/finding_db.py for persisted/normalized records.
4. Core processing pipeline
The main internal flow is:
input findings → validation/modeling → deduplication → severity/evaluation → output assembly.
core/deduplication.py is the key module for grouping or merging similar reports. core/evaluation.py appears responsible for scoring or classifying findings after deduplication.
5. LLM/model integration layer
core/claude_model.py and core/gemini_model.py indicate the service can delegate parts of triage to different AI providers. These modules form an abstraction boundary between the deterministic orchestration code and external model APIs.
6. Prompt and decision support
core/prompt_utils.py suggests prompt construction is separated from the model adapters. This keeps prompt logic centralized and reusable across providers, which matters if the same triage rules are sent to Claude or Gemini with different wrappers.
7. Persistence layer
database/mongodb_handler.py is the repository’s storage boundary. It likely stores findings, deduplication results, or workflow state in MongoDB. The presence of docker-compose.yml strongly suggests MongoDB is part of the local development/runtime stack.
8. Data models and contracts
models/finding_input.py and models/finding_db.py define the schema boundary between untrusted incoming report data and internal/persisted records. This separation is important for an architecture diagram because it shows where validation and normalization happen.
9. Orchestration and runtime behavior
task_utils.py and types.py imply some shared workflow coordination and typed domain objects. These likely connect API handlers, deduplication, model calls, and persistence into one execution path.
10. Configuration and deployment
config.py, .env.example, Dockerfile, and docker-compose.yml show the service is containerized and environment-driven. The repo also includes GitHub Actions workflows for ci-staging and ci-prod, indicating separate build/test/deploy paths for staging and production.
11. Quality and verification boundary
The test suite is split into unit and integration tests, which reveals the intended seams of the system:
- unit tests for prompt logic, deduplication, evaluation, cache, and task processing
- integration tests for API behavior, caching, database interaction, model integration, and end-to-end workflow
This suggests the architecture is designed around testable service boundaries rather than a monolith of tightly coupled code.
12. How to draw the diagram
A good architecture diagram would show:
external clients → API entrypoint (main.py) → validation/models → deduplication/evaluation core → optional LLM provider adapters (Claude/Gemini via prompt utilities) → MongoDB persistence → JSON response.
Cross-cutting concerns include configuration/env, containerization, and CI/CD workflows.
Main entry points: evaluation.py Severity scoring · integration Integration tests · task_utils.py Task utils · prompt_utils.py Prompt builder
Components 18
Service
Python triage service and its runtime modules
- API entry · Starts the service and exposes the request boundary
Demo/app/main.py - Task utils · Shared workflow helpers for triage execution
Demo/app/task_utils.py - Domain types · Shared typed objects used across the service
Demo/app/types.py - Input model · Validates and structures incoming finding payloads
Demo/app/models/finding_input.py - Persisted model · Normalized record shape used for storage and output
Demo/app/models/finding_db.py - Dedup engine · Groups and merges overlapping vulnerability reports
Demo/app/core/deduplication.py - Severity scoring · Classifies and scores consolidated findings
Demo/app/core/evaluation.py - Prompt builder · Constructs reusable prompts for model-backed triage
Demo/app/core/prompt_utils.py - Claude adapter · Provider wrapper for Claude-based triage steps
Demo/app/core/claude_model.py - Gemini adapter · Provider wrapper for Gemini-based triage steps
Demo/app/core/gemini_model.py - Mongo handler · MongoDB access for storing and retrieving triage data
Demo/app/database/mongodb_handler.py
Runtime
Container, config, and deployment boundaries
- Config · Runtime configuration and environment wiring
Demo/app/config.py - Dockerfile · Container image definition for the service
Demo/Dockerfile - Compose stack · Local service stack including MongoDB
Demo/docker-compose.yml - CI staging · GitHub Actions path for staging validation and delivery
.github/workflows/ci-staging.yaml - CI prod · GitHub Actions path for production validation and delivery
.github/workflows/ci-prod.yaml
Tests
Unit and integration verification boundaries
- Unit tests · Fast checks for deduplication, evaluation, prompts, and task logic
Demo/tests/unit - Integration tests · End-to-end checks for API, cache, database, models, and workflow behavior
Demo/tests/integration
Connections 29
- API entry → Input model: parse · API requests are converted into validated input findings
- API entry → Task utils: orchestrate · Main entrypoint coordinates the triage workflow
- Task utils → Dedup engine: dedupe · Workflow routes findings through deduplication
- Task utils → Severity scoring: score · Workflow applies severity or risk evaluation
- Task utils → Mongo handler: persist · Consolidated results can be stored in MongoDB
- Dedup engine → Persisted model: normalize · Merged results are shaped into persisted records
- Severity scoring → Persisted model: annotate · Scored findings are assembled into output records
- Dedup engine → Prompt builder: prompt · Dedup logic can prepare model-facing instructions
- Severity scoring → Prompt builder: prompt · Evaluation can reuse centralized prompt construction
- Prompt builder → Claude adapter: send · Prompts are executed through the Claude adapter
- Prompt builder → Gemini adapter: send · Prompts are executed through the Gemini adapter
- Claude adapter → Severity scoring: assist · Model output feeds triage classification
- Gemini adapter → Severity scoring: assist · Model output feeds triage classification
- Input model → Domain types: share types · Input schema relies on shared domain types
- Persisted model → Domain types: share types · Persisted schema relies on shared domain types
- API entry → Config: load · Service startup consumes environment configuration
- Mongo handler → Compose stack: backed by · Local runtime stack includes MongoDB service
- Dockerfile → Compose stack: build · Compose stack is built from the container image
- CI staging → Dockerfile: build · Staging pipeline builds the containerized service
- CI prod → Dockerfile: build · Production pipeline builds the containerized service
- Unit tests → Dedup engine: verify · Unit tests cover core deduplication behavior
- Unit tests → Severity scoring: verify · Unit tests cover scoring and classification logic
- Unit tests → Prompt builder: verify · Unit tests cover prompt construction
- Unit tests → Task utils: verify · Unit tests cover workflow helpers
- Integration tests → API entry: verify · Integration tests exercise the API boundary
- Integration tests → Mongo handler: verify · Integration tests cover database interaction
- Integration tests → Claude adapter: verify · Integration tests cover model-provider integration
- Integration tests → Gemini adapter: verify · Integration tests cover model-provider integration
- Integration tests → Input model: verify · Integration tests cover model parsing and validation