# ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation architecture

> Architecture diagram of the GitHub repository ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation
- Repository: https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation

## Overview

The product is a persona-driven chatbot paired with a multi-agent response evaluation workflow. A testing runner generates contextual questions, gets chatbot replies, and submits each question-answer pair to the evaluation graph; that graph extracts candidate facts, runs quality checks, and aggregates scores and fact-check results. The graph also preserves prompt construction, model configuration, and web-search fact checking. Source excerpts show the graph’s evaluator dispatch and model wrapper; runner wiring and some integrations are documented in the README rather than sampled source.

Main entry points: [`langgraph_evaluation_engine.py`](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py) (Fact extraction), [`llm_wrapper.py`](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/llm_wrapper.py) (LLM wrapper), [`test_evaluation.py`](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/test_evaluation.py) (Evaluation runner), [`bot_agent.py`](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/bot_agent.py) (Persona chatbot)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_chat["Chatbot"]
  node_bot["Persona chatbot<br/>[bot_agent.py]"]
  node_prompt["Prompt builder<br/>[prompt_builder.py]"]
end

subgraph group_evaluation["Evaluation workflow"]
  node_tester["Evaluation runner<br/>[test_evaluation.py]"]
  node_question_gen["Question generator"]
  node_engine["Evaluation graph"]
  node_fact_extract["Fact extraction"]
  node_clarity["Clarity evaluation"]
  node_pedagogy["Pedagogy evaluation"]
  node_statistics["Statistical evaluation"]
  node_fact_check["Fact checking"]
  node_tone["Tone evaluation"]
  node_aggregate["Result aggregation"]
  node_eval_tools["Evaluation tools"]
end

subgraph group_model["LLM access"]
  node_llm["LLM wrapper<br/>[llm_wrapper.py]"]
  node_config["Model configuration<br/>[config.py]"]
end

subgraph group_integrations["External services"]
  node_openrouter{{"OpenRouter API"}}
  node_search{{"DuckDuckGo search"}}
end

node_user(("Test initiator"))

node_user -->|"starts evaluation"| node_tester
node_tester -->|"requests questions"| node_question_gen
node_tester -->|"gets responses"| node_bot
node_tester -->|"submits turns"| node_engine
node_bot -->|"builds persona prompt"| node_prompt
node_bot -->|"requests response"| node_llm
node_question_gen -->|"generates questions"| node_llm
node_engine -->|"dispatches"| node_fact_extract
node_fact_extract -->|"dispatches"| node_clarity
node_fact_extract -->|"dispatches"| node_pedagogy
node_fact_extract -->|"dispatches"| node_statistics
node_fact_extract -->|"dispatches"| node_fact_check
node_fact_extract -->|"dispatches"| node_tone
node_clarity -->|"passes results"| node_aggregate
node_pedagogy -->|"passes results"| node_aggregate
node_statistics -->|"passes results"| node_aggregate
node_fact_check -->|"passes results"| node_aggregate
node_tone -->|"passes results"| node_aggregate
node_engine -->|"invokes evaluators"| node_eval_tools
node_eval_tools -->|"requests model analysis"| node_llm
node_fact_check -->|"searches claims"| node_search
node_llm -->|"loads model settings"| node_config
node_llm -->|"sends completions"| node_openrouter

click node_tester "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/test_evaluation.py"
click node_question_gen "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/question_generator.py"
click node_bot "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/bot_agent.py"
click node_prompt "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/prompt_builder.py"
click node_engine "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_fact_extract "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_clarity "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_pedagogy "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_statistics "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_fact_check "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_tone "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_aggregate "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py"
click node_eval_tools "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/evaluation_tools.py"
click node_llm "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/llm_wrapper.py"
click node_config "https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bot,node_prompt toneBlue
class node_tester,node_question_gen,node_engine,node_fact_extract,node_clarity,node_pedagogy,node_statistics,node_fact_check,node_tone,node_aggregate,node_eval_tools toneAmber
class node_llm,node_config toneMint
class node_openrouter,node_search toneRose
class node_user toneIndigo
```

## Components

### Chatbot

- [Persona chatbot](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/bot_agent.py): `bot_agent.py`
- [Prompt builder](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/prompt_builder.py): `prompt_builder.py`

### Evaluation workflow

- [Evaluation runner](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/test_evaluation.py): `test_evaluation.py`
- [Question generator](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/question_generator.py): `question_generator.py`
- [Evaluation graph](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Fact extraction](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Clarity evaluation](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Pedagogy evaluation](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Statistical evaluation](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Fact checking](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Tone evaluation](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Result aggregation](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/langgraph_evaluation_engine.py): `langgraph_evaluation_engine.py`
- [Evaluation tools](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/evaluation_tools.py): `evaluation_tools.py`

### LLM access

- [LLM wrapper](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/llm_wrapper.py): `llm_wrapper.py`
- [Model configuration](https://github.com/ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation/blob/main/config.py): `config.py`

### External services

- **OpenRouter API**
- **DuckDuckGo search**

### Other components

- **Test initiator**

## Connections

- Test initiator → Evaluation runner: starts evaluation (evidence: `README.md`)
- Evaluation runner → Question generator: requests questions (evidence: `README.md`)
- Evaluation runner → Persona chatbot: gets responses (evidence: `README.md`)
- Evaluation runner → Evaluation graph: submits turns (evidence: `README.md`)
- Persona chatbot → Prompt builder: builds persona prompt (evidence: `bot_agent.py`)
- Persona chatbot → LLM wrapper: requests response (evidence: `bot_agent.py`)
- Question generator → LLM wrapper: generates questions (evidence: `question_generator.py`)
- Evaluation graph → Fact extraction: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Fact extraction → Clarity evaluation: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Fact extraction → Pedagogy evaluation: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Fact extraction → Statistical evaluation: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Fact extraction → Fact checking: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Fact extraction → Tone evaluation: dispatches (evidence: `langgraph_evaluation_engine.py`)
- Clarity evaluation → Result aggregation: passes results (evidence: `langgraph_evaluation_engine.py`)
- Pedagogy evaluation → Result aggregation: passes results (evidence: `langgraph_evaluation_engine.py`)
- Statistical evaluation → Result aggregation: passes results (evidence: `langgraph_evaluation_engine.py`)
- Fact checking → Result aggregation: passes results (evidence: `langgraph_evaluation_engine.py`)
- Tone evaluation → Result aggregation: passes results (evidence: `langgraph_evaluation_engine.py`)
- Evaluation graph → Evaluation tools: invokes evaluators (evidence: `langgraph_evaluation_engine.py`)
- Evaluation tools → LLM wrapper: requests model analysis (evidence: `evaluation_tools.py`)
- Fact checking → DuckDuckGo search: searches claims (evidence: `README.md`)
- LLM wrapper → Model configuration: loads model settings (evidence: `llm_wrapper.py`)
- LLM wrapper → OpenRouter API: sends completions (evidence: `llm_wrapper.py`)
