How dnmohanty/data-recon-engine works
Crowdsourced submissions enter through the API, are queued, then normalized and embedded before the worker resolves them against PostgreSQL. The worker updates entity state transactionally and publishes committed changes for the dashboard’s SSE stream. The graph also includes the documented demo route and the separate similarity-threshold evaluation capability. Relationships involving runtime dependencies are shown where excerpts support them; database schema details are represented by init.sql, but its contents were not sampled.
Main entry points: worker.ts Ingest worker · index.ts Ingest and stream API · evaluate.ts Threshold evaluator · page.tsx Live dashboard
Components 15
Ingestion API
- Ingest and stream API
src/index.ts - Ingest job queue
Entity Resolution
- Ingest worker
src/worker.ts - Name normalization
src/normalize.ts - MiniLM embeddings
src/worker.ts - Match threshold
src/config.ts - Entity matching
src/worker.ts
State and Storage
- Reputation state machine
src/stateMachine.ts - PostgreSQL entities
init.sql - Event deduplication
src/worker.ts - Redis
Live Dashboard
- Live dashboard
frontend/src/app/page.tsx
Similarity Evaluation
- Threshold evaluator
src/evaluate.ts - Labeled name pairs
src/pairs.json
Other components
- Data submitter
Connections 17
- Data submitter → Live dashboard: submits events
frontend/src/app/page.tsx - Live dashboard → Ingest and stream API: sends ingest and demo
frontend/src/app/page.tsx - Ingest and stream API → Ingest job queue: enqueues jobs
src/index.ts - Ingest job queue → Ingest worker: dispatches jobs
src/worker.ts - Ingest worker → Name normalization: normalizes names
src/worker.ts - Ingest worker → MiniLM embeddings: computes embeddings
src/worker.ts - Ingest worker → Match threshold: uses threshold
src/worker.ts - Ingest worker → Event deduplication: records event IDs
src/worker.ts - Ingest worker → Entity matching: resolves entities
src/worker.ts - Entity matching → PostgreSQL entities: queries and updates
src/worker.ts - Entity matching → Reputation state machine: updates reputation
src/worker.ts - Ingest worker → Redis: publishes committed changes
src/worker.ts - Ingest and stream API → Redis: subscribes to updates
src/index.ts - Ingest and stream API → Live dashboard: streams SSE updates
src/index.ts - Threshold evaluator → Labeled name pairs: reads labeled pairs
src/evaluate.ts - Threshold evaluator → Name normalization: normalizes pair names
src/evaluate.ts - Threshold evaluator → MiniLM embeddings: computes pair embeddings
src/evaluate.ts