How dagster-io/dagster-open-platform works
Main Components
a. Data Sources Layer:
• External Services Integration
• GitHub (dlt/sources/github)
• Buildkite (dlt/sources/buildkite)
• HubSpot (dlt/sources/hubspot)
• Thinkific (dlt/sources/thinkific)
• Stripe (stripe/)
• PyPI (pypi/)
• AWS Services (aws/)
• Segment (segment/)
• Scout (scout/)
• Sling (sling/)
b. Data Integration Layer:
• ETL/ELT Tools
• Fivetran (fivetran/)
• DLT (Data Load Tool) (dlt/)
• Hightouch (hightouch/)
c. Data Warehouse Layer:
• Storage Systems
• Snowflake (snowflake/)
d. Transformation Layer:
• DBT Integration (dbt/)
• Models
• Snapshots
• Macros
• Tests
[External Data Sources]
• ↓
[Data Integration Layer]
• ↓
[Data Warehouse]
• ↓
[Transformation Layer]
• ↓
[Analytics & Operations]
Key Relationships to Highlight
• Orchestration role of Dagster across all components
• Data flow patterns between layers
• Scheduled jobs and sensors with distinct symbols
Architectural Patterns to Emphasize
• Modular design (separate directories for each integration)
• Resource abstraction (resources.py files)
• Schedule management (schedules.py files)
• Asset organization (assets.py files)
Additional Elements
• Annotations for key configuration points
• Critical paths and dependencies
• Error handling and monitoring points
Technical Details
• Asset definitions and dependencies
• Schedule and partition configurations
• Resource requirements and configurations
• Integration points with external systems
Remember to include:
• Version control integration points
• Testing frameworks
• Monitoring and observability components
• Error handling and recovery paths
• Data quality check points
This system design should emphasize Dagster's role as the central orchestrator while clearly showing how it integrates with various data sources, transformation tools, and storage systems in a production environment.