How hoglandets-it/beetl works
Main Components
a. Core Components:
• ETL Engine (Beetl Core)
• Configuration Handler
• Source Connectors
• Transformers
• Result Handler
b. Data Sources/Destinations (based on the /src/beetl/sources directory):
• SQL Databases (MySQL, PostgreSQL, SQL Server)
• MongoDB
• REST API
• XML
• CSV
• Static Data
• Faker (for testing)
• iTop
┌─ Configuration Layer ─────────────────────┐
• YAML/JSON/Python Dict Configuration │
• Column Definitions │
• Sync Rules │
• Core Processing Layer ──────────────────┤
• ┌─ Source Layer ┐ ┌─ Transform Layer ┐│
• Connectors │ -> │ - Data Frames ││
• Extractors │ │ - Type Convert ││
• ┘ └────────────────┬─┘│
• ┌─ Destination Layer ───────────────┐ │
• Target Systems │<──┘
• Data Loading │ │
• ┘ │
• ┘
Key Elements to Highlight
a. Source Connectors:
• Them by type (SQL, NoSQL, API, File-based)
• Interface inheritance from base connector
b. Transformation Pipeline:
• Data flow through transformers
• Supported transformation types:
• Frames
• Integer operations
• String operations
• Regular expressions
• Structural transformations
• iTop-specific transformations
c. Configuration Flow:
• How configuration is parsed
• Validation process
• Demonstrate how it connects to source/destination selection
Specific Implementation Details
Technical aspects:
• Polars integration for data processing
• Python class hierarchy
• Async/sync operation support
• Error handling flow
• Validation checkpoints
External Dependencies
Connections to:
• Polars for data processing
• Database drivers
• API clients
• XML processors
Testing Infrastructure
A separate section showing:
• Integration testing framework
• Benchmark testing setup
• Unit testing components
• Test containers
Deployment Options
Deployment possibilities:
• Docker container
• Kubernetes deployment
• Direct Python package installation
For this specific ETL tool, focus on showing the clear separation between configuration, extraction, transformation, and loading phases, while highlighting the extensibility points that allow for adding new source types and transformers.