How ioaviator/de_hackathon_usa_college_info works
Main Components
• Data Ingestion: A module that extracts data from the external USA College Scorecard API. (This is supported by parts of the ELT scripts in include/ELT/extract.py.)
• Data Transformation & ETL Pipeline: Represent the transformation and processing steps – including authentication, configuration, extraction, transformation, and loading. These are detailed in the include/ELT folder (modules like transform.py, load.py, and database utilities).
• Storage Layers:
• Raw Data Store: Azure Blob Storage is used to store raw/parquet files.
• Processed Data Store: Azure PostgreSQL holds the structured and analytic-ready data.
• Orchestration & Scheduling: Apache Airflow is used for orchestration (with DAGs defined in the dags directory plus the include/dag_context logic).
• Containerization and CI/CD:
• Docker: Dockerfile indicates containerization, with the project built into a container image.
• Azure Container Registry: Managed via Terraform configuration under terraform/container_registry.
• GitHub Actions: Automated testing, linting, Terraform deployment, and image build workflows (files under .github/workflows).
• Infrastructure as Code: Terraform scripts (located in terraform/) provisioning multiple Azure resources including the storage backend, PostgreSQL database, and container registry.
• Dev Environment and Deployment Tools:
• Astro CLI for Apache Airflow management.
• Python virtual environment setup and package management (requirements.txt, packages.txt).
Relationships and Data Flow
• The pipeline is triggered either manually (via main.py) or through Airflow orchestrated DAGs (from the dags directory).
• Data is extracted from the API, transformed by ETL modules and then:
• Stored in the raw data layer on Azure Blob Storage (for archival and later processing).
• Loaded into Azure PostgreSQL for analytics (visualized in Power BI as noted in the README).
• Docker and Terraform allow packaging of the application and automated infrastructure deployments.
• GitHub Actions continuously integrate code changes, perform linting/tests, build containers, and deploy using Terraform to Azure.
Architectural Patterns & Design Principles
• Separation of Concerns: The ETL process is split among several clear modules (auth, extract, transform, load, and database utilities).
• Infrastructure as Code: All cloud resource provisioning is automated via Terraform.
• Containerization & CI/CD: The use of Docker combined with GitHub Actions simplifies deployment and ensures consistency.
• Orchestration: Apache Airflow (managed with Astro CLI) orchestrates the workflow, allowing modularity and scheduling.
• Cloud-Native Elements: Integration with Azure services (Blob Storage, PostgreSQL, Container Registry) is central to the design.
Technologies and Frameworks to Label
• Python and its libraries for ETL processing.
• Apache Airflow for orchestration.
• Terraform for infra provisioning.
• Docker for containerization.
• Azure services: Blob Storage, PostgreSQL, Container Registry.
• GitHub Actions for continuous integration and deployment.