How otobi1/de-fi-analytics works
Main Components
a. Infrastructure Provisioning:
• Terraform: Represents the cloud infrastructure provisioning files (main.tf, outputs.tf, variables.tf) that set up key resources such as Cloud Build, Composer, GCS buckets, VM instances, and BigQuery datasets.
b. CI/CD and Deployment Pipeline:
• Cloud Build: Configured by cloudbuild.yaml, this component is responsible for building the project, fetching files from GitHub, and deploying them to the Composer environment.
c. Orchestration and Scheduling:
• Cloud Composer (Airflow): Represented by the “dags” folder that contains Airflow DAGs (e.g., de_fi_history_coingecko_ingest.py, de_fi_hourly_coingecko_ingest.py). This component schedules and manages the execution of the ingestion pipelines.
d. Data Ingestion and Processing:
• Ingestion Scripts: Located in the “scripts” directory (ingest.py, ingest_data.py, ingest_history.py, ingest_hourly.py), these scripts handle the actual data ingestion from Coingecko into GCS.
• Notebooks/Test Files: The .ipynb files (like check_ingest.ipynb, test_ingest.ipynb) and test scripts indicate exploratory testing and validation of the ingestion process.
e. Data Storage:
• Cloud Storage (GCS) and BigQuery: As described in the README, data is ingested into GCS with an eventual publish to a BigQuery dataset for analytics.
Relationships and Interactions
• Terraform provisions the necessary cloud resources, including Cloud Composer (which runs Airflow DAGs) and Cloud Build.
• Cloud Build fetches the project files from GitHub and deploys the ingestion workflows.
• Cloud Composer uses the DAG files to schedule and trigger the appropriate ingestion scripts.
• The ingestion scripts (located in the “scripts” directory) execute, pulling data from Coingecko, and ingesting it into GCS.
• Data once available in GCS may then be loaded into BigQuery, making it available for further analytics.
• Test files and notebooks support validating the ingestion process.
Architectural Patterns and Design Principles
• Modular Design: The separation of concerns is evident—the file tree divides infrastructure (Terraform), orchestration (dags), and data ingestion (scripts).
• Infrastructure as Code: Terraform manages all infrastructure provisioning.
• CI/CD Integration: Cloud Build automates the deployment of code changes to the Composer environment.
• ETL/ELT Pipeline: The project functions as an ingest pipeline (Extract and Load) from an external API (Coingecko) into cloud storage/warehouse systems.
Relevant Technologies
• Google Cloud Platform components: Cloud Build, Cloud Composer (Airflow), Cloud Storage (GCS), and BigQuery.
• Infrastructure as Code with Terraform.
• Python for ingestion scripts and DAG definitions.
• Dockerfile indicates containerization which may be used for running components in a consistent environment.
• Notebooks for testing and exploratory analysis.
Specific Elements
• Terraform (Infrastructure Provisioning)
• Cloud Build (CI/CD Pipeline)
• Cloud Composer/Airflow (Task Scheduling and Orchestration)
• Ingestion Scripts (Data Processing)
• Google Cloud Storage & BigQuery (Data Storage/Analytics)
Detail and Clarity
• It’s important that each component is broken down as much as possible to capture dependencies.