# datahub-project/datahub architecture

> Architecture diagram of the GitHub repository datahub-project/datahub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datahub-project/datahub
- Repository: https://github.com/datahub-project/datahub

## Overview

DataHub is an open-source metadata catalog for discovering, governing, and observing data assets. Its central workflow brings metadata from data platforms into a unified catalog through batch or real-time ingestion, then makes that context available for search, lineage, governance, and APIs. People use the web application; AI agents can use MCP and agent-context integrations. The graph emphasizes these documented product stages and capabilities. Several links reflect README-level architecture rather than inspected call sites; those are marked as unverified where appropriate.

Main entry points: [`entity-registry`](https://github.com/datahub-project/datahub/tree/master/entity-registry) (Metadata catalog), [`metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java`](https://github.com/datahub-project/datahub/blob/master/metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java) (GraphQL API), [`datahub-agent-context/src/datahub_agent_context/context.py`](https://github.com/datahub-project/datahub/blob/master/datahub-agent-context/src/datahub_agent_context/context.py) (Agent context), [`metadata-ingestion/src/datahub/ingestion/run/pipeline.py`](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/run/pipeline.py) (Ingestion framework)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Metadata ingestion"]
  node_ingestion["Ingestion framework<br/>[pipeline.py]"]
  node_actions["Event actions"]
  node_action_sources["Event sources"]
  node_event_filters["Event filters"]
end

subgraph group_catalog["Catalog services"]
  node_metadata_graph[("Metadata catalog")]
  node_entity_search["Entity search"]
  node_lineage["Lineage search"]
  node_timeseries["Time-series aspects"]
  node_query_service["Query service<br/>[QueryService.java]"]
  node_graphql["GraphQL API"]
  node_openapi["OpenAPI entities"]
end

subgraph group_experience["User interfaces"]
  node_web["Web application"]
  node_frontend_server["Frontend server<br/>[Application.java]"]
end

subgraph group_agents["Agent integrations"]
  node_agent_context["Agent context<br/>[context.py]"]
  node_mcp_tools["MCP tools"]
  node_agent_frameworks["Agent integrations<br/>[builder.py]"]
end

node_sources(("Data platforms"))
node_user(("Catalog user"))
node_external_agent(("AI agent"))

node_sources -->|"provide metadata"| node_ingestion
node_ingestion -->|"populate catalog"| node_metadata_graph
node_sources -.->|"emit events"| node_action_sources
node_action_sources -.->|"filter events"| node_event_filters
node_event_filters -.->|"route events"| node_actions
node_actions -.->|"update metadata"| node_metadata_graph
node_metadata_graph -.->|"search metadata"| node_entity_search
node_metadata_graph -->|"retrieve lineage"| node_lineage
node_metadata_graph -.->|"retrieve aspects"| node_timeseries
node_metadata_graph -->|"read and write"| node_query_service
node_entity_search -.->|"serve queries"| node_graphql
node_query_service -.->|"serve queries"| node_graphql
node_metadata_graph -.->|"serve entities"| node_openapi
node_user -->|"discover assets"| node_web
node_web -.->|"request application"| node_frontend_server
node_frontend_server -.->|"request catalog data"| node_graphql
node_external_agent -->|"request context"| node_agent_context
node_agent_context -.->|"expose tools"| node_mcp_tools
node_agent_context -.->|"integrate agents"| node_agent_frameworks
node_mcp_tools -.->|"query catalog"| node_graphql

click node_ingestion "https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/run/pipeline.py"
click node_actions "https://github.com/datahub-project/datahub/blob/master/datahub-actions/src/datahub_actions/pipeline/pipeline_manager.py"
click node_action_sources "https://github.com/datahub-project/datahub/tree/master/datahub-actions/src/datahub_actions/plugin/source"
click node_event_filters "https://github.com/datahub-project/datahub/tree/master/datahub-actions/src/datahub_actions/plugin/filter"
click node_metadata_graph "https://github.com/datahub-project/datahub/tree/master/entity-registry"
click node_entity_search "https://github.com/datahub-project/datahub/blob/master/metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java"
click node_lineage "https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java"
click node_timeseries "https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java"
click node_query_service "https://github.com/datahub-project/datahub/blob/master/metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java"
click node_graphql "https://github.com/datahub-project/datahub/blob/master/metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java"
click node_openapi "https://github.com/datahub-project/datahub/blob/master/metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/controller/GenericEntitiesController.java"
click node_web "https://github.com/datahub-project/datahub/blob/master/datahub-web-react/src/app/searchV2/useGetSearchQueryInputs.ts"
click node_frontend_server "https://github.com/datahub-project/datahub/blob/master/datahub-frontend/app/controllers/Application.java"
click node_agent_context "https://github.com/datahub-project/datahub/blob/master/datahub-agent-context/src/datahub_agent_context/context.py"
click node_mcp_tools "https://github.com/datahub-project/datahub/tree/master/datahub-agent-context/src/datahub_agent_context/mcp_tools"
click node_agent_frameworks "https://github.com/datahub-project/datahub/blob/master/datahub-agent-context/src/datahub_agent_context/langchain_tools/builder.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_actions,node_action_sources,node_event_filters,node_user toneBlue
class node_metadata_graph,node_entity_search,node_lineage,node_timeseries,node_query_service,node_graphql,node_openapi toneAmber
class node_web,node_frontend_server toneMint
class node_agent_context,node_mcp_tools,node_agent_frameworks toneRose
class node_sources,node_external_agent toneIndigo
```

## Components

### Metadata ingestion

- [Ingestion framework](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/run/pipeline.py): `metadata-ingestion/src/datahub/ingestion/run/pipeline.py`
- [Event actions](https://github.com/datahub-project/datahub/blob/master/datahub-actions/src/datahub_actions/pipeline/pipeline_manager.py): `datahub-actions/src/datahub_actions/pipeline/pipeline_manager.py`
- [Event sources](https://github.com/datahub-project/datahub/tree/master/datahub-actions/src/datahub_actions/plugin/source): `datahub-actions/src/datahub_actions/plugin/source`
- [Event filters](https://github.com/datahub-project/datahub/tree/master/datahub-actions/src/datahub_actions/plugin/filter): `datahub-actions/src/datahub_actions/plugin/filter`

### Catalog services

- [Metadata catalog](https://github.com/datahub-project/datahub/tree/master/entity-registry): `entity-registry`
- [Entity search](https://github.com/datahub-project/datahub/blob/master/metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java): `metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java`
- [Lineage search](https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java): `metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java`
- [Time-series aspects](https://github.com/datahub-project/datahub/blob/master/metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java): `metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java`
- [Query service](https://github.com/datahub-project/datahub/blob/master/metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java): `metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java`
- [GraphQL API](https://github.com/datahub-project/datahub/blob/master/metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java): `metadata-service/graphql-servlet-impl/src/main/java/com/datahub/graphql/GraphQLController.java`
- [OpenAPI entities](https://github.com/datahub-project/datahub/blob/master/metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/controller/GenericEntitiesController.java): `metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/controller/GenericEntitiesController.java`

### User interfaces

- [Web application](https://github.com/datahub-project/datahub/blob/master/datahub-web-react/src/app/searchV2/useGetSearchQueryInputs.ts): `datahub-web-react/src/app/searchV2/useGetSearchQueryInputs.ts`
- [Frontend server](https://github.com/datahub-project/datahub/blob/master/datahub-frontend/app/controllers/Application.java): `datahub-frontend/app/controllers/Application.java`

### Agent integrations

- [Agent context](https://github.com/datahub-project/datahub/blob/master/datahub-agent-context/src/datahub_agent_context/context.py): `datahub-agent-context/src/datahub_agent_context/context.py`
- [MCP tools](https://github.com/datahub-project/datahub/tree/master/datahub-agent-context/src/datahub_agent_context/mcp_tools): `datahub-agent-context/src/datahub_agent_context/mcp_tools`
- [Agent integrations](https://github.com/datahub-project/datahub/blob/master/datahub-agent-context/src/datahub_agent_context/langchain_tools/builder.py): `datahub-agent-context/src/datahub_agent_context/langchain_tools/builder.py`

### Other components

- **Data platforms**
- **Catalog user**
- **AI agent**

## Connections

- Data platforms → Ingestion framework: provide metadata (evidence: `README.md`)
- Ingestion framework → Metadata catalog: populate catalog (evidence: `README.md`)
- Data platforms → Event sources: emit events
- Event sources → Event filters: filter events
- Event filters → Event actions: route events
- Event actions → Metadata catalog: update metadata
- Metadata catalog → Entity search: search metadata (evidence: `metadata-service/services/src/main/java/com/linkedin/metadata/search/EntitySearchService.java`)
- Metadata catalog → Lineage search: retrieve lineage (evidence: `metadata-io/src/main/java/com/linkedin/metadata/search/LineageSearchService.java`)
- Metadata catalog → Time-series aspects: retrieve aspects (evidence: `metadata-io/src/main/java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectService.java`)
- Metadata catalog → Query service: read and write (evidence: `metadata-service/services/src/main/java/com/linkedin/metadata/service/QueryService.java`)
- Entity search → GraphQL API: serve queries
- Query service → GraphQL API: serve queries
- Metadata catalog → OpenAPI entities: serve entities
- Catalog user → Web application: discover assets (evidence: `README.md`)
- Web application → Frontend server: request application
- Frontend server → GraphQL API: request catalog data
- AI agent → Agent context: request context (evidence: `README.md`)
- Agent context → MCP tools: expose tools
- Agent context → Agent integrations: integrate agents
- MCP tools → GraphQL API: query catalog