How skalskibukowa/project-spark-scala-etl works
Main Components
• Data Sources:
• CSV files found in the Datasource folder.
• PostgreSQL databases (with Dockerized PostgreSQL setup in the PostgreSQL directory).
• Data Ingestion Layer:
• The Ingestions package (e.g., TransactionIngests.scala) that initiates the extraction of data.
• Data Cleaning and Transformation Layers:
• The cleaners package (e.g., TweetsCleaner, ForbesCleaner, TransactionCleaner, TweetsUserCleaner) which pre-process data.
• The analysers package (e.g., TransactionAnalyzer, TweetsAnalyzer, TweetsUserAnalyzer, ForbesAnalyzer, TransactionSearch) that perform further transformation, aggregation, or complex analysis.
• Data Loaders (Sink):
• The loaders package (e.g., ForbesLoader, TweetsLoader, TweetsUserLoader, TransactionLoader) that handle output to CSV, Parquet, or PostgreSQL.
• Supporting Components:
• The Docker setup (Dockerfile and docker-compose.yml) for running PostgreSQL.
• Build and dependency management with build.sbt and sbt configuration files.
• Unit tests (for verification) located under src/test.
Relationships and Data Flow
• Data flows from the sources (CSV files and PostgreSQL) into the ingestion layer.
• The ingested data is then passed to the cleaning modules to ensure quality.
• Clean data moves on to the analytical layer, where transformations and computations are applied.
• Finally, the processed data is passed to the loaders, which write the output into various sinks.
• External interactions such as Docker containers for PostgreSQL and any potential API endpoints if they exist.
Architectural Patterns and Design Principles
• Separation of Concerns: Clearly divide the ETL process into ingestion, cleaning, analysis, and loading stages.
• Modular Structure: Each functionality is encapsulated within its respective package, which aids in both maintainability and testing.
• Use of Docker for Database Environment: This enhances environment consistency and ease of deployment.
• Clear Data Pipeline Flow: Ensure that transformations follow a logical progression from extraction to loading.
Practical Steps for the Principal Software Engineer
• Zoom into each processing block (for Ingestion, Cleaning, Analysis, and Loading) if more detail is needed.
• That all components are clearly labeled with their package or module names (as evident in the file tree).
• Dependencies such as the transformation logic available in the analysers and cleaners, and the module responsible for interacting with external data sinks.