How bruin-data/ingestr works
Main Components
a) Source Layer:
• Sources into categories:
• Databases (PostgreSQL, BigQuery, Snowflake, etc.)
• Platforms/APIs (Adjust, Airtable, Facebook Ads, etc.)
• File Systems (CSV, S3)
b) Core Processing Layer:
• Command Line Interface (main.py)
• Source Handler (sources.py)
• Destination Handler (destinations.py)
• Factory Component (factory.py)
• Table Definition Manager (table_definition.py)
• Time Management (time.py)
• Telemetry System (telemetry/)
c) Destination Layer:
• Supported Databases (PostgreSQL, BigQuery, Snowflake, etc.)
Component Relationships
[Source Layer] → [Core Processing Layer] → [Destination Layer]
With these specific interactions:
• CLI receives user input (URIs and parameters)
• Factory creates appropriate source/destination handlers
• Source handlers extract data
• Core processing transforms and prepares data
• Destination handlers load data
Key Design Elements
• Rectangles: Core components
• Hexagons: Processing modules
• Blue: Source components
• Green: Core processing
• Orange: Destination components
• Gray: Supporting systems (telemetry, time management)
Implementation Details to Highlight
a) Source Handling:
[Source URI] → [Source Factory] → [Specific Source Handler] → [Data Extraction]
b) Data Flow:
[Extracted Data] → [Processing/Transformation] → [Loading Strategy (append/merge/delete+insert)] → [Destination]
Architectural Patterns to Emphasize
• Factory Pattern (for source/destination creation)
• Strategy Pattern (for different loading strategies)
• Adapter Pattern (for different source/destination implementations)