How rat-nick/film-data-ingestion-pipeline works
What this project is (type + purpose)
This is a data ingestion / ETL pipeline implemented as an Apache Airflow DAG, packaged to run locally via Docker Compose, and persisting data into a PostgreSQL database. The pipeline’s business goal is to discover films weekly from TMDB, transform them into a structured form, and load them into a “media” Postgres schema.
Key indicators from repo:
• dags/weekly_film_discovery_dag.py → orchestration layer (Airflow DAG definition).
• dags/utils/films.py → integration + transformation utilities (TMDB fetch, genres, shaping).
• docker-compose.yaml → runtime topology (Airflow services + Postgres).
• schema.sql → target data model in Postgres.
• .env with TMDB_API_KEY + Airflow UID/GID → runtime config/secrets.
Orchestration / Control Plane
• dags/weekly_film_discovery_dag.py
• Represents: Airflow DAG with scheduled trigger, task graph, retries, task dependencies, etc.
Integration + Transform Layer
• dags/utils/films.py
• Represents: TMDB client + transformation helpers (fetch films, fetch genres, normalize metadata).
Data Plane / Storage
• schema.sql
• Represents: Postgres schema (tables for production data; likely films, genres, mapping tables).
Deployment / Runtime Topology
• docker-compose.yaml
• Represents: containerized services (Airflow webserver/scheduler/worker(s) + Postgres DB).
• Also implies networking (service discovery like media-db) and ports (e.g., host 15432).
External Dependency
• TMDB API (not in tree, but in README and implied by TMDB_API_KEY)
Transient Storage
• README explicitly says: “Process and save the data to a temporary file” and “Clean up temporary files”
• Represents: ephemeral filesystem (likely within Airflow container volume or local path mounted into container)
What the README adds (runtime behaviors to capture)
The weekly run does:
Cleanup temp files
Also important operational facts:
• Airflow UI at localhost:8080 (Airflow webserver component)
• Airflow Connection ID media-metadata-db targeting Postgres host media-db, schema media
• Postgres exposed on host port 15432 (for exploration)
“Don’t overthink it”: separate into as many components as possible (but still accurate)
Given the repo, the “maximally separated but still faithful” component breakdown is: