How mongodb-partners/mdb-temporal-pra works
This repository is a Temporal-and-Atlas reference application for durable document ingestion and research chat. Its main path is an S3 object event, through a Temporal ingestion workflow that extracts, chunks, embeds, and indexes content, then a user question through the research API and durable agent to retrieval tools and an answer. A separate backfill path supports embedding-model changes and blue/green cutover. The UI exposes agent progress. Some subsystem links are based on README descriptions or SOURCE INDEX references; unsampled implementation details are not inferred.
Main entry points: ingest.py Ingest activities · ingest_workflow.py Ingest workflow · agent_workflow.py Research workflow · tools.py Agent tools
Components 22
Ingestion
- Ingestion trigger
pipeline/trigger.py - Ingest workflow
pipeline/workflows/ingest_workflow.py - Ingest activities
pipeline/activities/ingest.py - File extractors
pipeline/extractors - Chunk contracts
pipeline/models.py
Research agent
- Chat UI
agent/ui/src/App.tsx - Research API
agent/api.py - Research workflow
agent/agent_workflow.py - Agent tools
agent/tools.py - Vector retrieval
pipeline/retrieval.py
Index operations
- Backfill trigger
pipeline/trigger_backfill.py - Backfill workflow
pipeline/workflows/backfill_workflow.py - Backfill activities
pipeline/activities/backfill.py - Active-index cutover
pipeline/cutover.py - Active index pointer
pipeline/config_store.py
Data and AI services
- MongoDB Atlas
- S3 storage
- Voyage AI
- Temporal
- Hosted web search
Other components
- S3 object event
- Research user
Connections 26
- S3 object event → Ingestion trigger: starts ingestion
- Ingestion trigger → Ingest workflow: starts workflow
- Ingest workflow → Ingest activities: runs activities
pipeline/workflows/ingest_workflow.py - Ingest activities → File extractors: selects extractor
pipeline/activities/ingest.py - Ingest workflow → Chunk contracts: uses S3 reference
pipeline/workflows/ingest_workflow.py - Ingest activities → Chunk contracts: builds document IDs
pipeline/activities/ingest.py - Ingest activities → S3 storage: downloads objects
pipeline/activities/ingest.py - Ingest activities → MongoDB Atlas: stages and indexes chunks
pipeline/activities/ingest.py - Ingest activities → Voyage AI: embeds chunks
pipeline/activities/ingest.py - Research user → Chat UI: asks question
- Chat UI → Research API: starts and polls
agent/ui/src/App.tsx - Research API → Research workflow: starts workflow
- Research workflow → Agent tools: invokes tools
agent/agent_workflow.py - Agent tools → Vector retrieval: runs vector search
agent/tools.py - Vector retrieval → Active index pointer: reads active pointer
pipeline/retrieval.py - Vector retrieval → MongoDB Atlas: searches active collection
pipeline/retrieval.py - Agent tools → Voyage AI: reranks candidates
agent/tools.py - Research workflow → Hosted web search: uses web search
- Backfill trigger → Backfill workflow: starts backfill
pipeline/trigger_backfill.py - Backfill workflow → Backfill activities: runs activities
pipeline/workflows/backfill_workflow.py - Backfill activities → MongoDB Atlas: reads and writes chunks
pipeline/activities/backfill.py - Backfill activities → Voyage AI: re-embeds chunks
pipeline/activities/backfill.py - Active-index cutover → Active index pointer: flips active pointer
pipeline/cutover.py - Active index pointer → MongoDB Atlas: stores pointer
pipeline/config_store.py - Temporal → Ingest workflow: orchestrates
- Temporal → Research workflow: orchestrates