# dnmohanty/data-recon-engine architecture

> Architecture diagram of the GitHub repository dnmohanty/data-recon-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dnmohanty/data-recon-engine
- Repository: https://github.com/dnmohanty/data-recon-engine

## Overview

Crowdsourced submissions enter through the API, are queued, then normalized and embedded before the worker resolves them against PostgreSQL. The worker updates entity state transactionally and publishes committed changes for the dashboard’s SSE stream. The graph also includes the documented demo route and the separate similarity-threshold evaluation capability. Relationships involving runtime dependencies are shown where excerpts support them; database schema details are represented by init.sql, but its contents were not sampled.

Main entry points: [`src/worker.ts`](https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts) (Ingest worker), [`src/index.ts`](https://github.com/dnmohanty/data-recon-engine/blob/main/src/index.ts) (Ingest and stream API), [`src/evaluate.ts`](https://github.com/dnmohanty/data-recon-engine/blob/main/src/evaluate.ts) (Threshold evaluator), [`frontend/src/app/page.tsx`](https://github.com/dnmohanty/data-recon-engine/blob/main/frontend/src/app/page.tsx) (Live dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion API"]
  node_api["Ingest and stream API<br/>[index.ts]"]
  node_queue["Ingest job queue"]
end

subgraph group_resolution["Entity Resolution"]
  node_worker["Ingest worker<br/>[worker.ts]"]
  node_normalize["Name normalization<br/>[normalize.ts]"]
  node_embedding["MiniLM embeddings<br/>[worker.ts]"]
  node_threshold["Match threshold<br/>[config.ts]"]
  node_entity_match["Entity matching<br/>[worker.ts]"]
end

subgraph group_state["State and Storage"]
  node_state_machine["Reputation state machine<br/>[stateMachine.ts]"]
  node_postgres[("PostgreSQL entities<br/>[init.sql]")]
  node_processed["Event deduplication<br/>[worker.ts]"]
  node_redis[("Redis")]
end

subgraph group_live["Live Dashboard"]
  node_dashboard["Live dashboard<br/>[page.tsx]"]
end

subgraph group_evaluation["Similarity Evaluation"]
  node_evaluator["Threshold evaluator<br/>[evaluate.ts]"]
  node_pairs["Labeled name pairs<br/>[pairs.json]"]
end

node_submitter(("Data submitter"))

node_submitter -->|"submits events"| node_dashboard
node_dashboard -->|"sends ingest and demo"| node_api
node_api -->|"enqueues jobs"| node_queue
node_queue -->|"dispatches jobs"| node_worker
node_worker -->|"normalizes names"| node_normalize
node_worker -->|"computes embeddings"| node_embedding
node_worker -->|"uses threshold"| node_threshold
node_worker -->|"records event IDs"| node_processed
node_worker -->|"resolves entities"| node_entity_match
node_entity_match -->|"queries and updates"| node_postgres
node_entity_match -->|"updates reputation"| node_state_machine
node_worker -->|"publishes committed changes"| node_redis
node_api -->|"subscribes to updates"| node_redis
node_api -->|"streams SSE updates"| node_dashboard
node_evaluator -->|"reads labeled pairs"| node_pairs
node_evaluator -->|"normalizes pair names"| node_normalize
node_evaluator -->|"computes pair embeddings"| node_embedding

click node_api "https://github.com/dnmohanty/data-recon-engine/blob/main/src/index.ts"
click node_worker "https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts"
click node_normalize "https://github.com/dnmohanty/data-recon-engine/blob/main/src/normalize.ts"
click node_embedding "https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts"
click node_threshold "https://github.com/dnmohanty/data-recon-engine/blob/main/src/config.ts"
click node_entity_match "https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts"
click node_state_machine "https://github.com/dnmohanty/data-recon-engine/blob/main/src/stateMachine.ts"
click node_postgres "https://github.com/dnmohanty/data-recon-engine/blob/main/init.sql"
click node_processed "https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts"
click node_dashboard "https://github.com/dnmohanty/data-recon-engine/blob/main/frontend/src/app/page.tsx"
click node_evaluator "https://github.com/dnmohanty/data-recon-engine/blob/main/src/evaluate.ts"
click node_pairs "https://github.com/dnmohanty/data-recon-engine/blob/main/src/pairs.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_queue toneBlue
class node_worker,node_normalize,node_embedding,node_threshold,node_entity_match toneAmber
class node_state_machine,node_postgres,node_processed,node_redis toneMint
class node_dashboard toneRose
class node_evaluator,node_pairs,node_submitter toneIndigo
```

## Components

### Ingestion API

- [Ingest and stream API](https://github.com/dnmohanty/data-recon-engine/blob/main/src/index.ts): `src/index.ts`
- **Ingest job queue**

### Entity Resolution

- [Ingest worker](https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts): `src/worker.ts`
- [Name normalization](https://github.com/dnmohanty/data-recon-engine/blob/main/src/normalize.ts): `src/normalize.ts`
- [MiniLM embeddings](https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts): `src/worker.ts`
- [Match threshold](https://github.com/dnmohanty/data-recon-engine/blob/main/src/config.ts): `src/config.ts`
- [Entity matching](https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts): `src/worker.ts`

### State and Storage

- [Reputation state machine](https://github.com/dnmohanty/data-recon-engine/blob/main/src/stateMachine.ts): `src/stateMachine.ts`
- [PostgreSQL entities](https://github.com/dnmohanty/data-recon-engine/blob/main/init.sql): `init.sql`
- [Event deduplication](https://github.com/dnmohanty/data-recon-engine/blob/main/src/worker.ts): `src/worker.ts`
- **Redis**

### Live Dashboard

- [Live dashboard](https://github.com/dnmohanty/data-recon-engine/blob/main/frontend/src/app/page.tsx): `frontend/src/app/page.tsx`

### Similarity Evaluation

- [Threshold evaluator](https://github.com/dnmohanty/data-recon-engine/blob/main/src/evaluate.ts): `src/evaluate.ts`
- [Labeled name pairs](https://github.com/dnmohanty/data-recon-engine/blob/main/src/pairs.json): `src/pairs.json`

### Other components

- **Data submitter**

## Connections

- Data submitter → Live dashboard: submits events (evidence: `frontend/src/app/page.tsx`)
- Live dashboard → Ingest and stream API: sends ingest and demo (evidence: `frontend/src/app/page.tsx`)
- Ingest and stream API → Ingest job queue: enqueues jobs (evidence: `src/index.ts`)
- Ingest job queue → Ingest worker: dispatches jobs (evidence: `src/worker.ts`)
- Ingest worker → Name normalization: normalizes names (evidence: `src/worker.ts`)
- Ingest worker → MiniLM embeddings: computes embeddings (evidence: `src/worker.ts`)
- Ingest worker → Match threshold: uses threshold (evidence: `src/worker.ts`)
- Ingest worker → Event deduplication: records event IDs (evidence: `src/worker.ts`)
- Ingest worker → Entity matching: resolves entities (evidence: `src/worker.ts`)
- Entity matching → PostgreSQL entities: queries and updates (evidence: `src/worker.ts`)
- Entity matching → Reputation state machine: updates reputation (evidence: `src/worker.ts`)
- Ingest worker → Redis: publishes committed changes (evidence: `src/worker.ts`)
- Ingest and stream API → Redis: subscribes to updates (evidence: `src/index.ts`)
- Ingest and stream API → Live dashboard: streams SSE updates (evidence: `src/index.ts`)
- Threshold evaluator → Labeled name pairs: reads labeled pairs (evidence: `src/evaluate.ts`)
- Threshold evaluator → Name normalization: normalizes pair names (evidence: `src/evaluate.ts`)
- Threshold evaluator → MiniLM embeddings: computes pair embeddings (evidence: `src/evaluate.ts`)
