# rat-nick/film-data-ingestion-pipeline architecture

> Architecture diagram of the GitHub repository rat-nick/film-data-ingestion-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rat-nick/film-data-ingestion-pipeline
- Repository: https://github.com/rat-nick/film-data-ingestion-pipeline

## Overview

##### What this project is (type + purpose)

This is a **data ingestion / ETL pipeline** implemented as an **Apache Airflow DAG**, packaged to run locally via **Docker Compose**, and persisting data into a **PostgreSQL** database. The pipeline’s business goal is to **discover films weekly from TMDB**, transform them into a structured form, and **load them into a “media” Postgres schema**.

Key indicators from repo:
- `dags/weekly_film_discovery_dag.py` → orchestration layer (Airflow DAG definition).
- `dags/utils/films.py` → integration + transformation utilities (TMDB fetch, genres, shaping).
- `docker-compose.yaml` → runtime topology (Airflow services + Postgres).
- `schema.sql` → target data model in Postgres.
- `.env` with `TMDB_API_KEY` + Airflow UID/GID → runtime config/secrets.

##### Orchestration / Control Plane
- `dags/weekly_film_discovery_dag.py`
- Represents: **Airflow DAG** with scheduled trigger, task graph, retries, task dependencies, etc.

##### Integration + Transform Layer
- `dags/utils/films.py`
- Represents: **TMDB client + transformation helpers** (fetch films, fetch genres, normalize metadata).

##### Data Plane / Storage
- `schema.sql`
- Represents: **Postgres schema** (tables for production data; likely films, genres, mapping tables).

##### Deployment / Runtime Topology
- `docker-compose.yaml`
- Represents: **containerized services** (Airflow webserver/scheduler/worker(s) + Postgres DB).
- Also implies **networking** (service discovery like `media-db`) and ports (e.g., host `15432`).

##### External Dependency
- TMDB API (not in tree, but in README and implied by `TMDB_API_KEY`)

##### Transient Storage
- README explicitly says: “Process and save the data to a temporary file” and “Clean up temporary files”
- Represents: **ephemeral filesystem** (likely within Airflow container volume or local path mounted into container)

##### What the README adds (runtime behaviors to capture)

The weekly run does:
##### Cleanup temp files

Also important operational facts:
- Airflow UI at `localhost:8080` (Airflow webserver component)
- Airflow Connection ID `media-metadata-db` targeting Postgres host `media-db`, schema `media`
- Postgres exposed on host port `15432` (for exploration)

##### “Don’t overthink it”: separate into as many components as possible (but still accurate)

Given the repo, the “maximally separated but still faithful” component breakdown is:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================
    %% External Actors / Systems
    %% =========================
    Operator(["Operator (Human)"]):::external
    Browser(["Browser"]):::external
    TMDB(["TMDB API (External)"]):::external

    %% =========================
    %% Local Environment Boundary
    %% =========================
    subgraph "Local Docker Compose Environment"
        direction TB

        Compose["Docker Compose Topology"]:::platform

        subgraph "Apache Airflow (Containers)"
            direction TB
            Webserver["Airflow Webserver (UI :8080)"]:::airflow
            Scheduler["Airflow Scheduler (weekly schedule)"]:::airflow
            Worker["Airflow Worker / Executor (task runtime)"]:::airflow

            subgraph "DAG: weekly_film_discovery"
                direction TB
                DAGFile["weekly_film_discovery_dag.py (DAG definition)"]:::code

                T1["Task: Fetch films from TMDB"]:::task
                T2["Task: Transform/Normalize metadata"]:::task
                T3["Task: Stage to temp file"]:::task
                T4["Task: Load into Postgres (media schema)"]:::task
                T5["Task: Cleanup temp files"]:::task
            end

            Utils["films.py (TMDB client + transforms)"]:::code
        end

        subgraph "Storage (Containers / Volumes)"
            direction TB
            TempFS[("Ephemeral Temp File Storage")]:::temp
            Postgres[("PostgreSQL Service: media-db (:5432)")]:::db
            MediaSchema[("Schema: media (tables from schema.sql)")]:::db
        end

        subgraph "Configuration / Secrets"
            direction TB
            EnvNote[".env (TMDB_API_KEY, Airflow UID/GID)"]:::config
            ConnNote["Airflow Connection: media-metadata-db (host=media-db, schema=media)"]:::config
        end
    end

    Readme["README (ops: ports, run steps, temp file behavior)"]:::doc

    %% =========================
    %% Control / UI Flows
    %% =========================
    Operator -->|"uses"| Browser
    Browser -->|"opens UI"| Webserver
    Webserver -.->|"controls/monitors DAGs"| Scheduler

    %% =========================
    %% Scheduling / Orchestration
    %% =========================
    Scheduler -.->|"triggers weekly run"| DAGFile
    DAGFile -.->|"schedules tasks"| Worker

    %% =========================
    %% DAG Task Dependencies
    %% =========================
    T1 -->|"outputs raw discovery data"| T2
    T2 -->|"outputs structured film metadata"| T3
    T3 -->|"stage artifact path"| T4
    T4 -->|"load complete"| T5

    %% =========================
    %% Data Plane Interactions
    %% =========================
    Worker -->|"executes"| T1
    Worker -->|"executes"| T2
    Worker -->|"executes"| T3
    Worker -->|"executes"| T4
    Worker -->|"executes"| T5

    T1 -->|"HTTP(S) calls"| TMDB
    T2 -.->|"uses"| Utils

    T3 -->|"writes temp file"| TempFS
    T4 -->|"reads temp file"| TempFS
    T5 -->|"deletes temp file(s)"| TempFS

    T4 -->|"SQL INSERT/UPSERT via connection"| Postgres
    Postgres -->|"persists to"| MediaSchema

    %% =========================
    %% Configuration Consumption
    %% =========================
    EnvNote -.->|"provides TMDB_API_KEY"| T1
    ConnNote -.->|"provides DB creds/host/schema"| T4

    %% =========================
    %% Deployment / Docs Context
    %% =========================
    Compose -.->|"defines containers/network/ports"| Webserver
    Compose -.->|"defines containers/network/ports"| Scheduler
    Compose -.->|"defines containers/network/ports"| Worker
    Compose -.->|"defines service: media-db + port mapping host:15432->5432"| Postgres
    Readme -.->|"documents setup + behavior"| Compose

    %% =========================
    %% Click Events (from mapping)
    %% =========================
    click DAGFile "https://github.com/rat-nick/film-data-ingestion-pipeline/blob/main/dags/weekly_film_discovery_dag.py"
    click Utils "https://github.com/rat-nick/film-data-ingestion-pipeline/blob/main/dags/utils/films.py"
    click Compose "https://github.com/rat-nick/film-data-ingestion-pipeline/blob/main/docker-compose.yaml"
    click MediaSchema "https://github.com/rat-nick/film-data-ingestion-pipeline/blob/main/schema.sql"
    click Readme "https://github.com/rat-nick/film-data-ingestion-pipeline/blob/main/README.md"

    %% =========================
    %% Styles
    %% =========================
    classDef external fill:#ECEFF1,stroke:#455A64,stroke-width:1.5px,color:#263238
    classDef platform fill:#E0E0E0,stroke:#616161,stroke-width:1.5px,color:#212121
    classDef airflow fill:#E3F2FD,stroke:#1565C0,stroke-width:1.8px,color:#0D47A1
    classDef task fill:#BBDEFB,stroke:#1E88E5,stroke-width:1.6px,color:#0D47A1
    classDef code fill:#D1C4E9,stroke:#5E35B1,stroke-width:1.6px,color:#311B92
    classDef db fill:#C8E6C9,stroke:#2E7D32,stroke-width:1.8px,color:#1B5E20
    classDef temp fill:#DCEDC8,stroke:#558B2F,stroke-width:1.6px,color:#33691E
    classDef config fill:#FFF9C4,stroke:#F9A825,stroke-width:1.6px,color:#5D4037
    classDef doc fill:#FFE0B2,stroke:#EF6C00,stroke-width:1.6px,color:#4E342E
```
