How datahub-project/datahub works
DataHub is an open-source metadata catalog for discovering, governing, and observing data assets. Its central workflow brings metadata from data platforms into a unified catalog through batch or real-time ingestion, then makes that context available for search, lineage, governance, and APIs. People use the web application; AI agents can use MCP and agent-context integrations. The graph emphasizes these documented product stages and capabilities. Several links reflect README-level architecture rather than inspected call sites; those are marked as unverified where appropriate.
Main entry points: entity-registry Metadata catalog · GraphQLController.java GraphQL API · context.py Agent context · pipeline.py Ingestion framework
Components 19
Metadata ingestion
- Ingestion framework
metadata-ingestion/src/datahub/ingestion/run/pipeline.py - Event actions
datahub-actions/src/datahub_actions/pipeline/pipeline_manager.py - Event sources
datahub-actions/src/datahub_actions/plugin/source - Event filters
datahub-actions/src/datahub_actions/plugin/filter
Catalog services
- Metadata catalog
entity-registry - Entity search
metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java - Lineage search
metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java - Time-series aspects
metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java - Query service
metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java - GraphQL API
metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java - OpenAPI entities
metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/controller/GenericEntitiesController.java
User interfaces
- Web application
datahub-web-react/src/app/searchV2/useGetSearchQueryInputs.ts - Frontend server
datahub-frontend/app/controllers/Application.java
Agent integrations
- Agent context
datahub-agent-context/src/datahub_agent_context/context.py - MCP tools
datahub-agent-context/src/datahub_agent_context/mcp_tools - Agent integrations
datahub-agent-context/src/datahub_agent_context/langchain_tools/builder.py
Other components
- Data platforms
- Catalog user
- AI agent
Connections 20
- Data platforms → Ingestion framework: provide metadata
README.md - Ingestion framework → Metadata catalog: populate catalog
README.md - Data platforms → Event sources: emit events
- Event sources → Event filters: filter events
- Event filters → Event actions: route events
- Event actions → Metadata catalog: update metadata
- Metadata catalog → Entity search: search metadata
metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java - Metadata catalog → Lineage search: retrieve lineage
metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java - Metadata catalog → Time-series aspects: retrieve aspects
metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java - Metadata catalog → Query service: read and write
metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java - Entity search → GraphQL API: serve queries
- Query service → GraphQL API: serve queries
- Metadata catalog → OpenAPI entities: serve entities
- Catalog user → Web application: discover assets
README.md - Web application → Frontend server: request application
- Frontend server → GraphQL API: request catalog data
- AI agent → Agent context: request context
README.md - Agent context → MCP tools: expose tools
- Agent context → Agent integrations: integrate agents
- MCP tools → GraphQL API: query catalog