How itversity/data-engineering-spark works
This repository is designed as an extensive lab environment for teaching data engineering skills—including Python, SQL, Hadoop, Hive, and Spark—using hands-on exercises. The project supports two main environments: one for Python/SQL labs and a more resource‐intensive Hadoop/Spark lab. The material is organized around educational content (Jupyter notebooks and SQL scripts) and the infrastructure needed to run single node (and, optionally, multi-node) clusters through Docker and Terraform.
Project Type and Purpose
• The repository is a training/lab environment creation tool used for teaching data engineering concepts.
• It supplies preconfigured Docker images and compose files to set up two labs: one focused on Python/SQL with JupyterLab and Postgres, and another on Hadoop and Spark along with Hive metastore.
• Additionally, there is support for provisioning multi-cloud lab environments using Terraform.
Main Components
• User Interface: JupyterLab – where students access notebooks.
• Database: PostgreSQL – used in the Python/SQL lab.
• Big Data Cluster: Hadoop (HDFS, YARN, MapReduce), Hive, Spark 2 and Spark 3 – provided through a dedicated Docker container (e.g., “itvdelab”).
• Data Storage: Multiple data folders hold datasets (such as card decks, HR data, election results, NYSE data, retail_db, etc.) that are used as material and input for lab exercises.
• Materials: The “itversity-material” directory houses the educational notebooks (organized by topics such as SQL queries, basic programming, Python file I/O, web scraping, etc.), and additional “python-and-sql-solutions” for reference.
• Infrastructure Provisioning: Terraform directories (for AWS, GCP, and MS labs) and Docker Compose files to deploy the two lab environments.
Relationships and Interactions
• The user accesses the system via a web browser (pointing to the JupyterLab interface) on the host machine.
• Docker Compose manages the different containers that house the lab environments:
– In the Python/SQL lab, containers include JupyterLab and PostgreSQL.
– In the Hadoop/Spark lab, a separate container is configured with Hadoop, Hive, and Spark along with a metastore.
• Data flows from local data sources (mounted as volumes or included in the image) into the respective processing environments.
• Students interact with the notebooks provided in the “itversity-material” folder via JupyterLab, which in turn communicates with the backend databases and the Hadoop/Spark services.
• Terraform scripts are used to provision or manage lab environments (start and stop lab operations) on various cloud providers, highlighting integration with cloud infrastructure.
Architectural Patterns and Design Principles
• Containerization: The environment is entirely containerized using Docker and orchestrated via Docker Compose, ensuring separation of concerns between Python/SQL and Hadoop/Spark components.
• Modularity: There is a clear split between educational content (the material notebooks and scripts) and the infrastructure that supports them (the containers and orchestration scripts).
• Extensibility: Terraform configurations and modular directories indicate the project is designed to be expanded (e.g., adding new lab modules or scaling out to multi-node clusters).
• Infrastructure as Code (IaC): The presence of Terraform directories shows an approach that codifies the infrastructure needed for lab setup.
Elements
• “JupyterLab Container” (for running notebooks)
• “PostgreSQL Container”
• “Hadoop/Spark Container” (listing subcomponents such as HDFS, Yarn, Hive, Spark 2/3)
• “Educational Material” (itversity-material directory and structured notebook content)
• “Data Sets” (data directory showing different datasets)
• “Terraform Scripts” (representing management of AWS, GCP, and MS labs)
• From the user’s browser to the JupyterLab container.
• Between JupyterLab and backend databases or Spark services.