How kuyu12/scytale-exercise works
Main Components
a) Data Source Layer:
• GitHub API (external system)
• Connection with authentication (GITHUB_TOKEN)
b) ETL Pipeline Components:
• Extract Module (src/extract.py)
• Transform Module (src/transform.py)
• Load Module (src/load.py)
c) Storage Components:
• Raw Data Storage (JSON files)
• Processed Data Storage (Parquet files)
d) Processing Engine:
• Apache Spark cluster
[GitHub API] → [Extract Module] → [Raw JSON Storage] → [Transform Module (PySpark)] → [Load Module] → [Parquet Storage]
Additional Details
a) Infrastructure Layer:
• Docker container boundary
• Spark cluster configuration
b) Data Flow Annotations:
• JSON format for raw data
• Parquet format for processed data
• Schema transformation details
c) Control Flow:
• Error handling paths
• Logging paths
• Configuration management
Key Relationships to Highlight
• How the abstract provider pattern interfaces with the GitHub API
• Relationship between models and schemas
• Demonstrate how the ETL components interact with Spark
• Logging and error handling across all components
Technical Specifications to Note
• PySpark as the processing engine
• GitHub API as the data source
• JSON as intermediate storage
• Parquet as final storage
• Docker containerization
• Python 3.9+ runtime
This architecture should emphasize the modularity of the system, particularly how the provider pattern allows for potential extension to other git providers beyond GitHub, and how the ETL pipeline maintains clear separation between extraction, transformation, and loading phases.