How rajiesh/evals-compare works
This repository is a mechanistic-interpretability research assistant plus an evaluation-framework comparison. The implemented query path starts at the CLI, reaches the orchestrator, and routes to specialist agents that can search through an MCP client and server. The README also documents a planned research synthesizer and three evaluation frameworks; DeepEval has sampled runtime evidence, while the other evaluation implementations are minimal or unsampled. Some routing and integration details are not shown, so those relationships are omitted.
Main entry points: main.py Query orchestrator · feature_extraction/agent.py Feature specialist · circuits_analysis/agent.py Circuits specialist · mcp_client.py Search MCP client
Components 16
Assistant runtime
- CLI modes
src/cli.py - Query orchestrator
src/main.py - Runtime settings
src/config/settings.py
Research agents
- Feature specialist
src/agents/feature_extraction/agent.py - Feature prompts
src/agents/feature_extraction/prompts.py - Circuits specialist
src/agents/circuits_analysis/agent.py - Circuits prompts
src/agents/circuits_analysis/prompts.py - Research synthesizer
src/agents/research_synthesizer/agent.py
Search integration
- Search MCP client
src/tools/mcp_client.py - Google search server
mcp_servers/search_server/server.py
Evaluation
- DeepEval evaluator
evals/deepeval/evaluator.py - Evaluation runner
evals/deepeval/run_evaluation.py - TrueLens evaluator
evals/truelens - Pydantic Eval
evals/pydantic_eval
Other components
- Researcher
- Google Custom Search
Connections 12
- Researcher → CLI modes: submits query
README.md - Query orchestrator → CLI modes: parses modes
src/main.py - Query orchestrator → Feature specialist: routes queries
src/main.py - Query orchestrator → Circuits specialist: routes queries
src/main.py - Feature specialist → Feature prompts: uses instructions
src/agents/feature_extraction/agent.py - Circuits specialist → Circuits prompts: uses instructions
src/agents/circuits_analysis/agent.py - Feature specialist → Search MCP client: searches through
src/agents/feature_extraction/agent.py - Circuits specialist → Search MCP client: searches through
src/agents/circuits_analysis/agent.py - Search MCP client → Google search server: connects to
src/tools/mcp_client.py - Google search server → Google Custom Search: queries
README.md - Evaluation runner → DeepEval evaluator: runs suite
evals/deepeval/run_evaluation.py - DeepEval evaluator → Query orchestrator: evaluates responses
evals/deepeval/evaluator.py