How ajayjai30/multi-agent-system-for-finance-bot-development-and-evaluation works
The product is a persona-driven chatbot paired with a multi-agent response evaluation workflow. A testing runner generates contextual questions, gets chatbot replies, and submits each question-answer pair to the evaluation graph; that graph extracts candidate facts, runs quality checks, and aggregates scores and fact-check results. The graph also preserves prompt construction, model configuration, and web-search fact checking. Source excerpts show the graph’s evaluator dispatch and model wrapper; runner wiring and some integrations are documented in the README rather than sampled source.
Main entry points: langgraph_evaluation_engine.py Fact extraction · llm_wrapper.py LLM wrapper · test_evaluation.py Evaluation runner · bot_agent.py Persona chatbot
Components 18
Chatbot
- Persona chatbot
bot_agent.py - Prompt builder
prompt_builder.py
Evaluation workflow
- Evaluation runner
test_evaluation.py - Question generator
question_generator.py - Evaluation graph
langgraph_evaluation_engine.py - Fact extraction
langgraph_evaluation_engine.py - Clarity evaluation
langgraph_evaluation_engine.py - Pedagogy evaluation
langgraph_evaluation_engine.py - Statistical evaluation
langgraph_evaluation_engine.py - Fact checking
langgraph_evaluation_engine.py - Tone evaluation
langgraph_evaluation_engine.py - Result aggregation
langgraph_evaluation_engine.py - Evaluation tools
evaluation_tools.py
LLM access
- LLM wrapper
llm_wrapper.py - Model configuration
config.py
External services
- OpenRouter API
- DuckDuckGo search
Other components
- Test initiator
Connections 23
- Test initiator → Evaluation runner: starts evaluation
README.md - Evaluation runner → Question generator: requests questions
README.md - Evaluation runner → Persona chatbot: gets responses
README.md - Evaluation runner → Evaluation graph: submits turns
README.md - Persona chatbot → Prompt builder: builds persona prompt
bot_agent.py - Persona chatbot → LLM wrapper: requests response
bot_agent.py - Question generator → LLM wrapper: generates questions
question_generator.py - Evaluation graph → Fact extraction: dispatches
langgraph_evaluation_engine.py - Fact extraction → Clarity evaluation: dispatches
langgraph_evaluation_engine.py - Fact extraction → Pedagogy evaluation: dispatches
langgraph_evaluation_engine.py - Fact extraction → Statistical evaluation: dispatches
langgraph_evaluation_engine.py - Fact extraction → Fact checking: dispatches
langgraph_evaluation_engine.py - Fact extraction → Tone evaluation: dispatches
langgraph_evaluation_engine.py - Clarity evaluation → Result aggregation: passes results
langgraph_evaluation_engine.py - Pedagogy evaluation → Result aggregation: passes results
langgraph_evaluation_engine.py - Statistical evaluation → Result aggregation: passes results
langgraph_evaluation_engine.py - Fact checking → Result aggregation: passes results
langgraph_evaluation_engine.py - Tone evaluation → Result aggregation: passes results
langgraph_evaluation_engine.py - Evaluation graph → Evaluation tools: invokes evaluators
langgraph_evaluation_engine.py - Evaluation tools → LLM wrapper: requests model analysis
evaluation_tools.py - Fact checking → DuckDuckGo search: searches claims
README.md - LLM wrapper → Model configuration: loads model settings
llm_wrapper.py - LLM wrapper → OpenRouter API: sends completions
llm_wrapper.py