# elastic/connectors architecture

> Architecture diagram of the GitHub repository elastic/connectors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elastic/connectors
- Repository: https://github.com/elastic/connectors

## Overview

Elastic Connectors is a self-managed crawler service plus a reusable SDK for implementing source connectors. The principal workflow is service startup and scheduling, source-specific retrieval, framework-level processing, and synchronization into Elasticsearch; the service also manages access-control syncs, cleanup, connector records, and preflight compatibility checks. Source adapters are grouped by family rather than enumerated individually. Relationships involving unsampled service and adapter implementations are shown only where documented or marked unverified; the README provides the product-level workflow, not detailed call traces.

Main entry points: [`app/connectors_service/connectors/agent/service_manager.py`](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/service_manager.py) (Agent service manager), [`app/connectors_service/connectors/service_cli.py`](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/service_cli.py) (Service CLI), [`libs/connectors_sdk/connectors_sdk/source.py`](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/source.py) (Source framework), [`app/connectors_service/connectors/services/content_sync_job_execution.py`](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/content_sync_job_execution.py) (Content sync)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_service["Service orchestration"]
  node_cli["Service CLI<br/>[service_cli.py]"]
  node_preflight["Startup checks<br/>[preflight_check.py]"]
  node_scheduler["Job scheduling<br/>[job_scheduling.py]"]
  node_content_sync["Content sync"]
  node_access_sync["Access sync"]
  node_cleanup["Job cleanup<br/>[job_cleanup.py]"]
  node_service_manager["Agent service manager<br/>[service_manager.py]"]
  node_config["Service configuration<br/>[config.py]"]
  node_agent_records["Connector records"]
  node_service_logger["Service logging<br/>[logger.py]"]
end

subgraph group_sources["Source adapters"]
  node_source_adapters["Source connectors"]
  node_gitlab_client["GitLab client<br/>[client.py]"]
  node_microsoft_graph["Microsoft Graph<br/>[graph.py]"]
end

subgraph group_sdk["Connector framework"]
  node_sdk_source["Source framework<br/>[source.py]"]
  node_filtering["Filtering rules<br/>[validation.py]"]
  node_basic_rules["Basic rule logic<br/>[basic_rule.py]"]
  node_extraction["Content extraction"]
  node_sdk_logger["SDK logging<br/>[logger.py]"]
end

subgraph group_storage["Elasticsearch integration"]
  node_connector_index[("Connector records index<br/>[connectors.py]")]
  node_es_client["Elasticsearch client<br/>[client.py]"]
end

node_operator(("Operator or API user"))
node_elasticsearch[("Elasticsearch")]

node_operator -->|"starts service"| node_cli
node_cli -->|"runs checks"| node_preflight
node_cli -->|"loads configuration"| node_config
node_cli -->|"starts services"| node_service_manager
node_service_manager -->|"starts services"| node_scheduler
node_service_manager -->|"starts services"| node_content_sync
node_service_manager -->|"starts services"| node_access_sync
node_service_manager -->|"starts services"| node_cleanup
node_agent_records -->|"checks records"| node_connector_index
node_source_adapters -.->|"implements framework"| node_sdk_source
node_gitlab_client -->|"uses framework"| node_sdk_source
node_microsoft_graph -.->|"supports adapters"| node_source_adapters
node_sdk_source -->|"validates filters"| node_filtering
node_filtering -->|"uses rules"| node_basic_rules
node_sdk_source -->|"extracts content"| node_extraction
node_content_sync -.->|"retrieves source data"| node_source_adapters
node_content_sync -.->|"syncs content"| node_es_client
node_es_client -.->|"reads and writes"| node_elasticsearch
node_sdk_logger -->|"used by service"| node_service_logger

click node_cli "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/service_cli.py"
click node_preflight "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/preflight_check.py"
click node_scheduler "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/job_scheduling.py"
click node_content_sync "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/content_sync_job_execution.py"
click node_access_sync "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/access_control_sync_job_execution.py"
click node_cleanup "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/job_cleanup.py"
click node_service_manager "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/service_manager.py"
click node_config "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/config.py"
click node_agent_records "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/connector_record_manager.py"
click node_service_logger "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/logger.py"
click node_source_adapters "https://github.com/elastic/connectors/tree/main/app/connectors_service/connectors/sources"
click node_gitlab_client "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/sources/gitlab/client.py"
click node_microsoft_graph "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/sources/shared/microsoft/graph.py"
click node_sdk_source "https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/source.py"
click node_filtering "https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/filtering/validation.py"
click node_basic_rules "https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/filtering/basic_rule.py"
click node_extraction "https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/content_extraction.py"
click node_sdk_logger "https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/logger.py"
click node_connector_index "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/protocol/connectors.py"
click node_es_client "https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_preflight,node_scheduler,node_content_sync,node_access_sync,node_cleanup,node_service_manager,node_config,node_agent_records,node_service_logger,node_operator toneBlue
class node_source_adapters,node_gitlab_client,node_microsoft_graph,node_elasticsearch toneAmber
class node_sdk_source,node_filtering,node_basic_rules,node_extraction,node_sdk_logger toneMint
class node_connector_index,node_es_client toneRose
```

## Components

### Service orchestration

- [Service CLI](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/service_cli.py): `app/connectors_service/connectors/service_cli.py`
- [Startup checks](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/preflight_check.py): `app/connectors_service/connectors/preflight_check.py`
- [Job scheduling](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/job_scheduling.py): `app/connectors_service/connectors/services/job_scheduling.py`
- [Content sync](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/content_sync_job_execution.py): `app/connectors_service/connectors/services/content_sync_job_execution.py`
- [Access sync](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/access_control_sync_job_execution.py): `app/connectors_service/connectors/services/access_control_sync_job_execution.py`
- [Job cleanup](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/services/job_cleanup.py): `app/connectors_service/connectors/services/job_cleanup.py`
- [Agent service manager](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/service_manager.py): `app/connectors_service/connectors/agent/service_manager.py`
- [Service configuration](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/config.py): `app/connectors_service/connectors/config.py`
- [Connector records](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/connector_record_manager.py): `app/connectors_service/connectors/agent/connector_record_manager.py`
- [Service logging](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/agent/logger.py): `app/connectors_service/connectors/agent/logger.py`

### Source adapters

- [Source connectors](https://github.com/elastic/connectors/tree/main/app/connectors_service/connectors/sources): `app/connectors_service/connectors/sources`
- [GitLab client](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/sources/gitlab/client.py): `app/connectors_service/connectors/sources/gitlab/client.py`
- [Microsoft Graph](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/sources/shared/microsoft/graph.py): `app/connectors_service/connectors/sources/shared/microsoft/graph.py`

### Connector framework

- [Source framework](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/source.py): `libs/connectors_sdk/connectors_sdk/source.py`
- [Filtering rules](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/filtering/validation.py): `libs/connectors_sdk/connectors_sdk/filtering/validation.py`
- [Basic rule logic](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/filtering/basic_rule.py): `libs/connectors_sdk/connectors_sdk/filtering/basic_rule.py`
- [Content extraction](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/content_extraction.py): `libs/connectors_sdk/connectors_sdk/content_extraction.py`
- [SDK logging](https://github.com/elastic/connectors/blob/main/libs/connectors_sdk/connectors_sdk/logger.py): `libs/connectors_sdk/connectors_sdk/logger.py`

### Elasticsearch integration

- [Connector records index](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/protocol/connectors.py): `app/connectors_service/connectors/protocol/connectors.py`
- [Elasticsearch client](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/client.py): `app/connectors_service/connectors/es/client.py`

### Other components

- **Operator or API user**
- **Elasticsearch**

## Connections

- Operator or API user → Service CLI: starts service (evidence: `README.md`)
- Service CLI → Startup checks: runs checks (evidence: `app/connectors_service/connectors/service_cli.py`)
- Service CLI → Service configuration: loads configuration (evidence: `app/connectors_service/connectors/service_cli.py`)
- Service CLI → Agent service manager: starts services (evidence: `app/connectors_service/connectors/service_cli.py`)
- Agent service manager → Job scheduling: starts services (evidence: `app/connectors_service/connectors/agent/service_manager.py`)
- Agent service manager → Content sync: starts services (evidence: `app/connectors_service/connectors/agent/service_manager.py`)
- Agent service manager → Access sync: starts services (evidence: `app/connectors_service/connectors/agent/service_manager.py`)
- Agent service manager → Job cleanup: starts services (evidence: `app/connectors_service/connectors/agent/service_manager.py`)
- Connector records → Connector records index: checks records (evidence: `app/connectors_service/connectors/agent/connector_record_manager.py`)
- Source connectors → Source framework: implements framework (evidence: `app/connectors_service/connectors/sources/gitlab/client.py`)
- GitLab client → Source framework: uses framework (evidence: `app/connectors_service/connectors/sources/gitlab/client.py`)
- Microsoft Graph → Source connectors: supports adapters (evidence: `app/connectors_service/connectors/sources/outlook_cloud/client.py`)
- Source framework → Filtering rules: validates filters (evidence: `libs/connectors_sdk/connectors_sdk/source.py`)
- Filtering rules → Basic rule logic: uses rules (evidence: `libs/connectors_sdk/connectors_sdk/filtering/validation.py`)
- Source framework → Content extraction: extracts content (evidence: `libs/connectors_sdk/connectors_sdk/source.py`)
- Content sync → Source connectors: retrieves source data
- Content sync → Elasticsearch client: syncs content
- Elasticsearch client → Elasticsearch: reads and writes
- SDK logging → Service logging: used by service (evidence: `app/connectors_service/connectors/agent/logger.py`)
