# rajiesh/evals-compare architecture

> Architecture diagram of the GitHub repository rajiesh/evals-compare, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajiesh/evals-compare
- Repository: https://github.com/rajiesh/evals-compare

## Overview

This repository is a mechanistic-interpretability research assistant plus an evaluation-framework comparison. The implemented query path starts at the CLI, reaches the orchestrator, and routes to specialist agents that can search through an MCP client and server. The README also documents a planned research synthesizer and three evaluation frameworks; DeepEval has sampled runtime evidence, while the other evaluation implementations are minimal or unsampled. Some routing and integration details are not shown, so those relationships are omitted.

Main entry points: [`src/main.py`](https://github.com/rajiesh/evals-compare/blob/main/src/main.py) (Query orchestrator), [`src/agents/feature_extraction/agent.py`](https://github.com/rajiesh/evals-compare/blob/main/src/agents/feature_extraction/agent.py) (Feature specialist), [`src/agents/circuits_analysis/agent.py`](https://github.com/rajiesh/evals-compare/blob/main/src/agents/circuits_analysis/agent.py) (Circuits specialist), [`src/tools/mcp_client.py`](https://github.com/rajiesh/evals-compare/blob/main/src/tools/mcp_client.py) (Search MCP client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["Assistant runtime"]
  node_cli["CLI modes<br/>[cli.py]"]
  node_orchestrator["Query orchestrator<br/>[main.py]"]
  node_settings["Runtime settings<br/>[settings.py]"]
end

subgraph group_research["Research agents"]
  node_feature["Feature specialist<br/>[agent.py]"]
  node_feature_prompts["Feature prompts<br/>[prompts.py]"]
  node_circuits["Circuits specialist<br/>[agent.py]"]
  node_circuits_prompts["Circuits prompts<br/>[prompts.py]"]
  node_synthesizer["Research synthesizer<br/>[agent.py]"]
end

subgraph group_search["Search integration"]
  node_mcp_client["Search MCP client<br/>[mcp_client.py]"]
  node_search_server["Google search server<br/>[server.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_deepeval["DeepEval evaluator<br/>[evaluator.py]"]
  node_eval_runner["Evaluation runner<br/>[run_evaluation.py]"]
  node_truelens["TrueLens evaluator"]
  node_pydantic_eval["Pydantic Eval"]
end

node_user(("Researcher"))
node_google_search["Google Custom Search"]

node_user -->|"submits query"| node_cli
node_orchestrator -->|"parses modes"| node_cli
node_orchestrator -->|"routes queries"| node_feature
node_orchestrator -->|"routes queries"| node_circuits
node_feature -->|"uses instructions"| node_feature_prompts
node_circuits -->|"uses instructions"| node_circuits_prompts
node_feature -->|"searches through"| node_mcp_client
node_circuits -->|"searches through"| node_mcp_client
node_mcp_client -->|"connects to"| node_search_server
node_search_server -->|"queries"| node_google_search
node_eval_runner -->|"runs suite"| node_deepeval
node_deepeval -->|"evaluates responses"| node_orchestrator

click node_cli "https://github.com/rajiesh/evals-compare/blob/main/src/cli.py"
click node_orchestrator "https://github.com/rajiesh/evals-compare/blob/main/src/main.py"
click node_settings "https://github.com/rajiesh/evals-compare/blob/main/src/config/settings.py"
click node_feature "https://github.com/rajiesh/evals-compare/blob/main/src/agents/feature_extraction/agent.py"
click node_feature_prompts "https://github.com/rajiesh/evals-compare/blob/main/src/agents/feature_extraction/prompts.py"
click node_circuits "https://github.com/rajiesh/evals-compare/blob/main/src/agents/circuits_analysis/agent.py"
click node_circuits_prompts "https://github.com/rajiesh/evals-compare/blob/main/src/agents/circuits_analysis/prompts.py"
click node_synthesizer "https://github.com/rajiesh/evals-compare/blob/main/src/agents/research_synthesizer/agent.py"
click node_mcp_client "https://github.com/rajiesh/evals-compare/blob/main/src/tools/mcp_client.py"
click node_search_server "https://github.com/rajiesh/evals-compare/blob/main/mcp_servers/search_server/server.py"
click node_deepeval "https://github.com/rajiesh/evals-compare/blob/main/evals/deepeval/evaluator.py"
click node_eval_runner "https://github.com/rajiesh/evals-compare/blob/main/evals/deepeval/run_evaluation.py"
click node_truelens "https://github.com/rajiesh/evals-compare/tree/main/evals/truelens"
click node_pydantic_eval "https://github.com/rajiesh/evals-compare/tree/main/evals/pydantic_eval"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_orchestrator,node_settings toneBlue
class node_feature,node_feature_prompts,node_circuits,node_circuits_prompts,node_synthesizer toneAmber
class node_mcp_client,node_search_server toneMint
class node_deepeval,node_eval_runner,node_truelens,node_pydantic_eval toneRose
class node_user,node_google_search toneIndigo
```

## Components

### Assistant runtime

- [CLI modes](https://github.com/rajiesh/evals-compare/blob/main/src/cli.py): `src/cli.py`
- [Query orchestrator](https://github.com/rajiesh/evals-compare/blob/main/src/main.py): `src/main.py`
- [Runtime settings](https://github.com/rajiesh/evals-compare/blob/main/src/config/settings.py): `src/config/settings.py`

### Research agents

- [Feature specialist](https://github.com/rajiesh/evals-compare/blob/main/src/agents/feature_extraction/agent.py): `src/agents/feature_extraction/agent.py`
- [Feature prompts](https://github.com/rajiesh/evals-compare/blob/main/src/agents/feature_extraction/prompts.py): `src/agents/feature_extraction/prompts.py`
- [Circuits specialist](https://github.com/rajiesh/evals-compare/blob/main/src/agents/circuits_analysis/agent.py): `src/agents/circuits_analysis/agent.py`
- [Circuits prompts](https://github.com/rajiesh/evals-compare/blob/main/src/agents/circuits_analysis/prompts.py): `src/agents/circuits_analysis/prompts.py`
- [Research synthesizer](https://github.com/rajiesh/evals-compare/blob/main/src/agents/research_synthesizer/agent.py): `src/agents/research_synthesizer/agent.py`

### Search integration

- [Search MCP client](https://github.com/rajiesh/evals-compare/blob/main/src/tools/mcp_client.py): `src/tools/mcp_client.py`
- [Google search server](https://github.com/rajiesh/evals-compare/blob/main/mcp_servers/search_server/server.py): `mcp_servers/search_server/server.py`

### Evaluation

- [DeepEval evaluator](https://github.com/rajiesh/evals-compare/blob/main/evals/deepeval/evaluator.py): `evals/deepeval/evaluator.py`
- [Evaluation runner](https://github.com/rajiesh/evals-compare/blob/main/evals/deepeval/run_evaluation.py): `evals/deepeval/run_evaluation.py`
- [TrueLens evaluator](https://github.com/rajiesh/evals-compare/tree/main/evals/truelens): `evals/truelens`
- [Pydantic Eval](https://github.com/rajiesh/evals-compare/tree/main/evals/pydantic_eval): `evals/pydantic_eval`

### Other components

- **Researcher**
- **Google Custom Search**

## Connections

- Researcher → CLI modes: submits query (evidence: `README.md`)
- Query orchestrator → CLI modes: parses modes (evidence: `src/main.py`)
- Query orchestrator → Feature specialist: routes queries (evidence: `src/main.py`)
- Query orchestrator → Circuits specialist: routes queries (evidence: `src/main.py`)
- Feature specialist → Feature prompts: uses instructions (evidence: `src/agents/feature_extraction/agent.py`)
- Circuits specialist → Circuits prompts: uses instructions (evidence: `src/agents/circuits_analysis/agent.py`)
- Feature specialist → Search MCP client: searches through (evidence: `src/agents/feature_extraction/agent.py`)
- Circuits specialist → Search MCP client: searches through (evidence: `src/agents/circuits_analysis/agent.py`)
- Search MCP client → Google search server: connects to (evidence: `src/tools/mcp_client.py`)
- Google search server → Google Custom Search: queries (evidence: `README.md`)
- Evaluation runner → DeepEval evaluator: runs suite (evidence: `evals/deepeval/run_evaluation.py`)
- DeepEval evaluator → Query orchestrator: evaluates responses (evidence: `evals/deepeval/evaluator.py`)
