How elastic/connectors works
Elastic Connectors is a self-managed crawler service plus a reusable SDK for implementing source connectors. The principal workflow is service startup and scheduling, source-specific retrieval, framework-level processing, and synchronization into Elasticsearch; the service also manages access-control syncs, cleanup, connector records, and preflight compatibility checks. Source adapters are grouped by family rather than enumerated individually. Relationships involving unsampled service and adapter implementations are shown only where documented or marked unverified; the README provides the product-level workflow, not detailed call traces.
Main entry points: service_manager.py Agent service manager · service_cli.py Service CLI · source.py Source framework · content_sync_job_execution.py Content sync
Components 22
Service orchestration
- Service CLI
app/connectors_service/connectors/service_cli.py - Startup checks
app/connectors_service/connectors/preflight_check.py - Job scheduling
app/connectors_service/connectors/services/job_scheduling.py - Content sync
app/connectors_service/connectors/services/content_sync_job_execution.py - Access sync
app/connectors_service/connectors/services/access_control_sync_job_execution.py - Job cleanup
app/connectors_service/connectors/services/job_cleanup.py - Agent service manager
app/connectors_service/connectors/agent/service_manager.py - Service configuration
app/connectors_service/connectors/config.py - Connector records
app/connectors_service/connectors/agent/connector_record_manager.py - Service logging
app/connectors_service/connectors/agent/logger.py
Source adapters
- Source connectors
app/connectors_service/connectors/sources - GitLab client
app/connectors_service/connectors/sources/gitlab/client.py - Microsoft Graph
app/connectors_service/connectors/sources/shared/microsoft/graph.py
Connector framework
- Source framework
libs/connectors_sdk/connectors_sdk/source.py - Filtering rules
libs/connectors_sdk/connectors_sdk/filtering/validation.py - Basic rule logic
libs/connectors_sdk/connectors_sdk/filtering/basic_rule.py - Content extraction
libs/connectors_sdk/connectors_sdk/content_extraction.py - SDK logging
libs/connectors_sdk/connectors_sdk/logger.py
Elasticsearch integration
- Connector records index
app/connectors_service/connectors/protocol/connectors.py - Elasticsearch client
app/connectors_service/connectors/es/client.py
Other components
- Operator or API user
- Elasticsearch
Connections 19
- Operator or API user → Service CLI: starts service
README.md - Service CLI → Startup checks: runs checks
app/connectors_service/connectors/service_cli.py - Service CLI → Service configuration: loads configuration
app/connectors_service/connectors/service_cli.py - Service CLI → Agent service manager: starts services
app/connectors_service/connectors/service_cli.py - Agent service manager → Job scheduling: starts services
app/connectors_service/connectors/agent/service_manager.py - Agent service manager → Content sync: starts services
app/connectors_service/connectors/agent/service_manager.py - Agent service manager → Access sync: starts services
app/connectors_service/connectors/agent/service_manager.py - Agent service manager → Job cleanup: starts services
app/connectors_service/connectors/agent/service_manager.py - Connector records → Connector records index: checks records
app/connectors_service/connectors/agent/connector_record_manager.py - Source connectors → Source framework: implements framework
app/connectors_service/connectors/sources/gitlab/client.py - GitLab client → Source framework: uses framework
app/connectors_service/connectors/sources/gitlab/client.py - Microsoft Graph → Source connectors: supports adapters
app/connectors_service/connectors/sources/outlook_cloud/client.py - Source framework → Filtering rules: validates filters
libs/connectors_sdk/connectors_sdk/source.py - Filtering rules → Basic rule logic: uses rules
libs/connectors_sdk/connectors_sdk/filtering/validation.py - Source framework → Content extraction: extracts content
libs/connectors_sdk/connectors_sdk/source.py - Content sync → Source connectors: retrieves source data
- Content sync → Elasticsearch client: syncs content
- Elasticsearch client → Elasticsearch: reads and writes
- SDK logging → Service logging: used by service
app/connectors_service/connectors/agent/logger.py