How lingabarani/dataextraction works
DataExtraction is an enterprise business-data collection and lead-intelligence platform. Its principal workflow gathers records from federal, geographic, and registry sources, coordinates extraction and transformation, then stores and exposes enriched organizations and leads through the API and dashboard. Distinct capabilities also include live discovery, chat-based search, AI-assisted orchestration, quality/governance, and exports. The graph uses the supplied source index and excerpts for implementation links; several README-described subsystems are represented by their exact tree paths, but links to unsampled implementations are omitted or marked unverified.
Main entry points: workflows Extraction workflows · pipeline_activities.py Pipeline activities · workers Temporal workers · chatbot Chat search
Components 27
User Experience
- Operations dashboard
frontend/src/app/(dashboard) - Search experience
frontend/src/app/(dashboard)/search - Chat search
packages/chatbot - Data export
frontend/src/components/export/export-manager.tsx
API Access
- FastAPI service
packages/api/main.py - Discovery API
packages/api/routers/discovery_router.py - Orchestration API
packages/api/routers/orchestration_router.py
Data Collection
- Federal registries
packages/scrapers/federal - Google Maps extraction
packages/scrapers/google_maps - OpenStreetMap extraction
packages/scrapers/open_map - Mexico registries
packages/scrapers/mexico - Live discovery
packages/discovery - Source ingestion coordinator
packages/pipeline/pipeline_orchestrator.py
Intelligence Pipeline
- Pipeline activities
packages/temporal/activities/pipeline_activities.py - Normalize and resolve
packages/pipeline/normalizer.py - Identity and quality
packages/pipeline/graph_identity_resolver.py - Contact validation
packages/pipeline/contact_validator.py - Lead building
packages/domain/lead_builder.py
Orchestration and Storage
- Extraction workflows
packages/temporal/workflows - Temporal workers
packages/temporal/workers - Autonomy agents
packages/agents/orchestration_agents - Storage activities
packages/temporal/activities/storage_activities.py - Knowledge graph ingestion
packages/infrastructure/kg_ingestion_bridge.py - PostgreSQL data store
packages/infrastructure/db_connector.py - Object lakehouse
packages/infrastructure/storage_lakehouse.py - LLM provider routing
packages/infrastructure/llm/router.py
Other components
- Platform user
Connections 23
- Platform user → Operations dashboard: operates
- Platform user → Search experience: searches
- Operations dashboard → FastAPI service: uses API
- Search experience → Discovery API: queries
- Discovery API → Live discovery: dispatches discovery
- Orchestration API → Temporal workers: controls workflows
- Temporal workers → Extraction workflows: runs
packages/temporal/workers/worker_search.py - Source ingestion coordinator → Federal registries: loads scrapers
packages/pipeline/pipeline_orchestrator.py - Extraction workflows → Federal registries: runs extraction
- Extraction workflows → Google Maps extraction: runs extraction
- Extraction workflows → OpenStreetMap extraction: runs extraction
- Extraction workflows → Mexico registries: runs extraction
- Pipeline activities → Normalize and resolve: calls
packages/temporal/activities/pipeline_activities.py - Pipeline activities → Identity and quality: calls
packages/temporal/activities/pipeline_activities.py - Pipeline activities → Contact validation: calls
packages/temporal/activities/pipeline_activities.py - Pipeline activities → Lead building: calls
packages/temporal/activities/pipeline_activities.py - Storage activities → PostgreSQL data store: reads and writes
packages/temporal/activities/storage_activities.py - Storage activities → Knowledge graph ingestion: ingests records
packages/temporal/activities/storage_activities.py - Knowledge graph ingestion → Object lakehouse: uses storage
packages/infrastructure/kg_ingestion_bridge.py - Chat search → PostgreSQL data store: searches records
packages/chatbot/integrations/ai_query_client.py - Chat search → LLM provider routing: uses language model
packages/chatbot/integrations/llm_client.py - Autonomy agents → Temporal workers: starts workflows
- Data export → FastAPI service: requests exports