How adityasingh0912/grabon-agentic-challenge works
1. What this repo is
- This is an autonomous merchant-onboarding document agent.
- Its job is to take in KYC-like document text, extract structured fields, verify them against tools/mock APIs, cross-check consistency, and output an onboarding decision with confidence.
2. Top-level orchestration
- main.py is the primary entry point for running the agent on a single document, a demo set, or a batch file.
- run_eval.py and eval/eval.py provide a separate evaluation path that runs the same agent logic against a fixed fixture set and scores the outputs.
- The orchestration layer is responsible for loading inputs, invoking the agent loop, and collecting final decisions/reports.
3. Core state model
- agent/state.py holds the central document state object, described in the README as a Pydantic model.
- This state is the shared contract between preprocessing, planning, tool execution, observation, and final decision-making.
- It likely carries extracted fields, verification results, reasoning history, budget counters, confidence, and final status.
4. Agent loop as the main control boundary
- agent/loop.py is the heart of the system.
- It implements the PAOD-style cycle: pre-process, plan, act, observe, decide.
- This is where budget enforcement, retry handling, and phase transitions happen; it is the main architectural boundary between orchestration and actual document-processing logic.
5. Deterministic vs LLM-driven work
- The repo intentionally separates cheap deterministic work from LLM work.
- Deterministic stages handle OCR cleanup, regex extraction, field merging, expiry checks, and cross-reference validation.
- LLMs are reserved for planning which tools to call next and for the final onboarding decision, which matches the README’s cost/control philosophy.
6. Tool subsystem
- tools/registry.py is the abstraction layer for all executable capabilities.
- Tools are registered dynamically and discovered at runtime rather than hardcoded into the loop.
- The tool set includes extraction, verification, cross-checking, and expiry logic; some tools are deterministic while others simulate unstable external services such as GST/PAN/bank verification.
7. External verification and failure handling
- The tool layer acts as the boundary to mock external dependencies.
- The architecture explicitly expects failures such as timeouts, rate limits, server errors, invalid input, and missing tools.
- Retry/backoff behavior is part of the control plane, not the tools themselves, which keeps the tool interface simple and the recovery policy centralized in the agent loop.
8. LLM routing layer
- agent/router.py selects which model/provider to use for each LLM-backed phase.
- The README indicates cost-aware routing across multiple providers, with cheaper models for planning and stronger models for final decisions.
- Architecturally, this is a policy layer that sits between the loop and the model APIs, allowing provider fallback without changing the agent logic.
9. Observability and operator feedback
- observability/terminal_ui.py provides a rich terminal dashboard for live runs.
- It surfaces current phase, budget progress, extracted fields, tool call history, retries, and the final decision.
- This is an operator-facing view rather than part of the decision pipeline, but it is important for understanding runtime behavior and diagnosing failures.
10. Evaluation pipeline
- The eval/ directory contains the offline/online scoring harness and fixtures.
- eval/fixtures/documents.json defines representative documents and expected outcomes.
- eval/eval.py runs the agent against the fixtures and computes correctness and safety metrics, making evaluation a first-class subsystem rather than an afterthought.
11. UI and server surfaces
- ui/index.html suggests a minimal browser-facing UI, likely for presenting or embedding run results.
- server.py likely exposes a local service boundary for the UI or for triggering runs over HTTP.
- These are peripheral interfaces; the core architecture remains the agent loop plus tool registry.
12. Operational constraints
- Budget enforcement is a key architectural control: step count, tool calls, tokens, wall-clock time, and cost all cap execution.
- This means the system is designed for bounded autonomy, not open-ended reasoning.
- In diagram terms, the budget checker is a guardrail around every loop iteration and every retry path.
13. End-to-end data flow
- Input document text enters through main.py or the eval harness.
- The document is normalized and stored in the shared state.
- The loop plans tool usage, executes tools, folds observations back into state, and repeats until budget exhaustion or completion.
- The final decision and confidence are emitted to the caller, with observability and eval consumers reading the same underlying run state.
14. How to draw the architecture
- Model this repo as three layers: orchestration/eval at the top, the agent loop in the middle, and tools/LLM providers at the bottom.
- Put agent/state.py as the shared data backbone spanning the middle layer.
- Show observability and eval as sidecars that read the same state and outputs, not as part of the core decision path.
Main entry points: loop.py Agent Loop · state.py State Model · registry.py Tool Registry · terminal_ui.py Terminal UI
Components 18
Orchestration
Entry points that load inputs, run the agent, and collect results.
Agent Core
Shared state and the control loop that drive document processing.
- Agent Loop · PAOD-style cycle: preprocess, plan, act, observe, decide.
agent/loop.py - State Model · Central document state shared across all phases.
agent/state.py - Budget Guard · Execution limits and retry boundaries enforced by the loop.
- Preprocess · Normalization, cleanup, and field preparation before planning.
- Planner · Chooses the next action or tool call.
- Observe · Folds tool outputs back into shared state.
- Decision · Produces the final onboarding outcome and confidence.
Execution Layer
Tool registry and model routing used by the loop to act and decide.
- Tool Registry · Runtime discovery and dispatch for executable capabilities.
tools/registry.py - Deterministic Tools · Regex extraction, merge, expiry checks, and cross-reference logic.
- Mock Verifiers · Simulated GST, PAN, and bank verification services.
- LLM Router · Selects models/providers with cost-aware fallback.
agent/router.py
Sidecars
Evaluation, observability, and lightweight UI/server surfaces.
- Terminal UI · Live dashboard for phases, budgets, tool history, and results.
observability/terminal_ui.py - Eval Harness · Runs fixtures through the agent and computes metrics.
eval/eval.py - Fixtures · Representative documents and expected outcomes.
eval/fixtures/documents.json - Web UI · Minimal browser-facing results surface.
ui/index.html - HTTP Server · Local service boundary for UI or run triggering.
server.py
Connections 20
- Main → Agent Loop: runs · Main loads input and invokes the core agent loop.
- Eval CLI → Eval Harness: runs · Eval CLI feeds fixtures into the scoring harness.
- Eval Harness → Fixtures: loads · Evaluation uses fixture documents and expected results.
- Agent Loop → State Model: reads/writes · The loop mutates the shared document state throughout execution.
- Agent Loop → Budget Guard: checks · Every iteration is bounded by budget and retry policy.
- Agent Loop → Preprocess: starts · Processing begins with deterministic preprocessing.
- Preprocess → State Model: updates · Preprocessing normalizes and seeds the state model.
- Agent Loop → Planner: plans · The loop asks the planner for the next action.
- Planner → LLM Router: routes through · Planning and decision phases select models via the router.
- Agent Loop → Tool Registry: dispatches · Planned actions are resolved through the tool registry.
- Tool Registry → Deterministic Tools: executes · Registry dispatches deterministic extraction and validation tools.
- Tool Registry → Mock Verifiers: calls · Registry also fronts mock external verification services.
- Agent Loop → Observe: observes · Tool outputs are folded back into state after each action.
- Observe → State Model: merges into · Observations enrich extracted fields and verification results.
- Agent Loop → Decision: decides · Final decision is produced once the run completes or stops.
- Decision → LLM Router: routes through · The final decision can use a stronger model/provider.
- Terminal UI → Agent Loop: watches · Terminal UI reads the live run state for display.
- Terminal UI → State Model: reads · Dashboard surfaces extracted fields, retries, and confidence.
- HTTP Server → Web UI: serves · Server exposes the browser UI surface.
- HTTP Server → Agent Loop: triggers · Server can initiate or proxy agent runs.