# lingabarani/dataextraction architecture

> Architecture diagram of the GitHub repository lingabarani/dataextraction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lingabarani/dataextraction
- Repository: https://github.com/lingabarani/dataextraction

## Overview

DataExtraction is an enterprise business-data collection and lead-intelligence platform. Its principal workflow gathers records from federal, geographic, and registry sources, coordinates extraction and transformation, then stores and exposes enriched organizations and leads through the API and dashboard. Distinct capabilities also include live discovery, chat-based search, AI-assisted orchestration, quality/governance, and exports. The graph uses the supplied source index and excerpts for implementation links; several README-described subsystems are represented by their exact tree paths, but links to unsampled implementations are omitted or marked unverified.

Main entry points: [`packages/temporal/workflows`](https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workflows) (Extraction workflows), [`packages/temporal/activities/pipeline_activities.py`](https://github.com/lingabarani/dataextraction/blob/main/packages/temporal/activities/pipeline_activities.py) (Pipeline activities), [`packages/temporal/workers`](https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workers) (Temporal workers), [`packages/chatbot`](https://github.com/lingabarani/dataextraction/tree/main/packages/chatbot) (Chat search)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User Experience"]
  node_dashboard["Operations dashboard"]
  node_search_ui["Search experience"]
  node_chat["Chat search"]
  node_export["Data export<br/>[export-manager.tsx]"]
end

subgraph group_access["API Access"]
  node_api["FastAPI service<br/>[main.py]"]
  node_discovery_api["Discovery API"]
  node_orchestration_api["Orchestration API"]
end

subgraph group_collection["Data Collection"]
  node_federal_sources["Federal registries"]
  node_maps_sources["Google Maps extraction"]
  node_openmap_sources["OpenStreetMap extraction"]
  node_mexico_sources["Mexico registries"]
  node_discovery["Live discovery"]
  node_source_coordinator["Source ingestion coordinator"]
end

subgraph group_processing["Intelligence Pipeline"]
  node_pipeline_activities["Pipeline activities"]
  node_normalization["Normalize and resolve<br/>[normalizer.py]"]
  node_identity_quality["Identity and quality"]
  node_contact_validation["Contact validation"]
  node_lead_builder["Lead building<br/>[lead_builder.py]"]
end

subgraph group_platform["Orchestration and Storage"]
  node_extract_workflows["Extraction workflows"]
  node_workers["Temporal workers"]
  node_agent_planner["Autonomy agents"]
  node_storage_activities["Storage activities"]
  node_kg_bridge["Knowledge graph ingestion"]
  node_database[("PostgreSQL data store<br/>[db_connector.py]")]
  node_lakehouse[("Object lakehouse")]
  node_llm["LLM provider routing<br/>[router.py]"]
end

node_operator(("Platform user"))

node_operator -.->|"operates"| node_dashboard
node_operator -.->|"searches"| node_search_ui
node_dashboard -.->|"uses API"| node_api
node_search_ui -.->|"queries"| node_discovery_api
node_discovery_api -.->|"dispatches discovery"| node_discovery
node_orchestration_api -.->|"controls workflows"| node_workers
node_workers -.->|"runs"| node_extract_workflows
node_source_coordinator -->|"loads scrapers"| node_federal_sources
node_extract_workflows -.->|"runs extraction"| node_federal_sources
node_extract_workflows -.->|"runs extraction"| node_maps_sources
node_extract_workflows -.->|"runs extraction"| node_openmap_sources
node_extract_workflows -.->|"runs extraction"| node_mexico_sources
node_pipeline_activities -->|"calls"| node_normalization
node_pipeline_activities -->|"calls"| node_identity_quality
node_pipeline_activities -->|"calls"| node_contact_validation
node_pipeline_activities -->|"calls"| node_lead_builder
node_storage_activities -->|"reads and writes"| node_database
node_storage_activities -->|"ingests records"| node_kg_bridge
node_kg_bridge -->|"uses storage"| node_lakehouse
node_chat -->|"searches records"| node_database
node_chat -->|"uses language model"| node_llm
node_agent_planner -.->|"starts workflows"| node_workers
node_export -.->|"requests exports"| node_api

click node_dashboard "https://github.com/lingabarani/dataextraction/tree/main/frontend/src/app/(dashboard)"
click node_search_ui "https://github.com/lingabarani/dataextraction/tree/main/frontend/src/app/(dashboard)/search"
click node_api "https://github.com/lingabarani/dataextraction/blob/main/packages/api/main.py"
click node_discovery_api "https://github.com/lingabarani/dataextraction/blob/main/packages/api/routers/discovery_router.py"
click node_orchestration_api "https://github.com/lingabarani/dataextraction/blob/main/packages/api/routers/orchestration_router.py"
click node_federal_sources "https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/federal"
click node_maps_sources "https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/google_maps"
click node_openmap_sources "https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/open_map"
click node_mexico_sources "https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/mexico"
click node_discovery "https://github.com/lingabarani/dataextraction/tree/main/packages/discovery"
click node_extract_workflows "https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workflows"
click node_workers "https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workers"
click node_agent_planner "https://github.com/lingabarani/dataextraction/tree/main/packages/agents/orchestration_agents"
click node_source_coordinator "https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/pipeline_orchestrator.py"
click node_pipeline_activities "https://github.com/lingabarani/dataextraction/blob/main/packages/temporal/activities/pipeline_activities.py"
click node_normalization "https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/normalizer.py"
click node_identity_quality "https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/graph_identity_resolver.py"
click node_contact_validation "https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/contact_validator.py"
click node_lead_builder "https://github.com/lingabarani/dataextraction/blob/main/packages/domain/lead_builder.py"
click node_storage_activities "https://github.com/lingabarani/dataextraction/blob/main/packages/temporal/activities/storage_activities.py"
click node_kg_bridge "https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/kg_ingestion_bridge.py"
click node_database "https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/db_connector.py"
click node_lakehouse "https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/storage_lakehouse.py"
click node_chat "https://github.com/lingabarani/dataextraction/tree/main/packages/chatbot"
click node_export "https://github.com/lingabarani/dataextraction/blob/main/frontend/src/components/export/export-manager.tsx"
click node_llm "https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/llm/router.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_search_ui,node_chat,node_export,node_operator toneBlue
class node_api,node_discovery_api,node_orchestration_api toneAmber
class node_federal_sources,node_maps_sources,node_openmap_sources,node_mexico_sources,node_discovery,node_source_coordinator toneMint
class node_pipeline_activities,node_normalization,node_identity_quality,node_contact_validation,node_lead_builder toneRose
class node_extract_workflows,node_workers,node_agent_planner,node_storage_activities,node_kg_bridge,node_database,node_lakehouse,node_llm toneIndigo
```

## Components

### User Experience

- [Operations dashboard](https://github.com/lingabarani/dataextraction/tree/main/frontend/src/app/(dashboard)): `frontend/src/app/(dashboard)`
- [Search experience](https://github.com/lingabarani/dataextraction/tree/main/frontend/src/app/(dashboard)/search): `frontend/src/app/(dashboard)/search`
- [Chat search](https://github.com/lingabarani/dataextraction/tree/main/packages/chatbot): `packages/chatbot`
- [Data export](https://github.com/lingabarani/dataextraction/blob/main/frontend/src/components/export/export-manager.tsx): `frontend/src/components/export/export-manager.tsx`

### API Access

- [FastAPI service](https://github.com/lingabarani/dataextraction/blob/main/packages/api/main.py): `packages/api/main.py`
- [Discovery API](https://github.com/lingabarani/dataextraction/blob/main/packages/api/routers/discovery_router.py): `packages/api/routers/discovery_router.py`
- [Orchestration API](https://github.com/lingabarani/dataextraction/blob/main/packages/api/routers/orchestration_router.py): `packages/api/routers/orchestration_router.py`

### Data Collection

- [Federal registries](https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/federal): `packages/scrapers/federal`
- [Google Maps extraction](https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/google_maps): `packages/scrapers/google_maps`
- [OpenStreetMap extraction](https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/open_map): `packages/scrapers/open_map`
- [Mexico registries](https://github.com/lingabarani/dataextraction/tree/main/packages/scrapers/mexico): `packages/scrapers/mexico`
- [Live discovery](https://github.com/lingabarani/dataextraction/tree/main/packages/discovery): `packages/discovery`
- [Source ingestion coordinator](https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/pipeline_orchestrator.py): `packages/pipeline/pipeline_orchestrator.py`

### Intelligence Pipeline

- [Pipeline activities](https://github.com/lingabarani/dataextraction/blob/main/packages/temporal/activities/pipeline_activities.py): `packages/temporal/activities/pipeline_activities.py`
- [Normalize and resolve](https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/normalizer.py): `packages/pipeline/normalizer.py`
- [Identity and quality](https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/graph_identity_resolver.py): `packages/pipeline/graph_identity_resolver.py`
- [Contact validation](https://github.com/lingabarani/dataextraction/blob/main/packages/pipeline/contact_validator.py): `packages/pipeline/contact_validator.py`
- [Lead building](https://github.com/lingabarani/dataextraction/blob/main/packages/domain/lead_builder.py): `packages/domain/lead_builder.py`

### Orchestration and Storage

- [Extraction workflows](https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workflows): `packages/temporal/workflows`
- [Temporal workers](https://github.com/lingabarani/dataextraction/tree/main/packages/temporal/workers): `packages/temporal/workers`
- [Autonomy agents](https://github.com/lingabarani/dataextraction/tree/main/packages/agents/orchestration_agents): `packages/agents/orchestration_agents`
- [Storage activities](https://github.com/lingabarani/dataextraction/blob/main/packages/temporal/activities/storage_activities.py): `packages/temporal/activities/storage_activities.py`
- [Knowledge graph ingestion](https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/kg_ingestion_bridge.py): `packages/infrastructure/kg_ingestion_bridge.py`
- [PostgreSQL data store](https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/db_connector.py): `packages/infrastructure/db_connector.py`
- [Object lakehouse](https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/storage_lakehouse.py): `packages/infrastructure/storage_lakehouse.py`
- [LLM provider routing](https://github.com/lingabarani/dataextraction/blob/main/packages/infrastructure/llm/router.py): `packages/infrastructure/llm/router.py`

### Other components

- **Platform user**

## Connections

- Platform user → Operations dashboard: operates
- Platform user → Search experience: searches
- Operations dashboard → FastAPI service: uses API
- Search experience → Discovery API: queries
- Discovery API → Live discovery: dispatches discovery
- Orchestration API → Temporal workers: controls workflows
- Temporal workers → Extraction workflows: runs (evidence: `packages/temporal/workers/worker_search.py`)
- Source ingestion coordinator → Federal registries: loads scrapers (evidence: `packages/pipeline/pipeline_orchestrator.py`)
- Extraction workflows → Federal registries: runs extraction
- Extraction workflows → Google Maps extraction: runs extraction
- Extraction workflows → OpenStreetMap extraction: runs extraction
- Extraction workflows → Mexico registries: runs extraction
- Pipeline activities → Normalize and resolve: calls (evidence: `packages/temporal/activities/pipeline_activities.py`)
- Pipeline activities → Identity and quality: calls (evidence: `packages/temporal/activities/pipeline_activities.py`)
- Pipeline activities → Contact validation: calls (evidence: `packages/temporal/activities/pipeline_activities.py`)
- Pipeline activities → Lead building: calls (evidence: `packages/temporal/activities/pipeline_activities.py`)
- Storage activities → PostgreSQL data store: reads and writes (evidence: `packages/temporal/activities/storage_activities.py`)
- Storage activities → Knowledge graph ingestion: ingests records (evidence: `packages/temporal/activities/storage_activities.py`)
- Knowledge graph ingestion → Object lakehouse: uses storage (evidence: `packages/infrastructure/kg_ingestion_bridge.py`)
- Chat search → PostgreSQL data store: searches records (evidence: `packages/chatbot/integrations/ai_query_client.py`)
- Chat search → LLM provider routing: uses language model (evidence: `packages/chatbot/integrations/llm_client.py`)
- Autonomy agents → Temporal workers: starts workflows
- Data export → FastAPI service: requests exports
