# kpdata-sci/apollo-audio architecture

> Architecture diagram of the GitHub repository kpdata-sci/apollo-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kpdata-sci/apollo-audio
- Repository: https://github.com/kpdata-sci/apollo-audio

## Overview

Apollo is a SoundCloud scraper and track library. Users discover curated playlists or submit playlist/profile URLs; the API queues scrape jobs, fetches and parses SoundCloud pages, then writes immutable raw JSON to the lake before upserting tracks into Postgres. The web UI browses, searches, plays and favorites stored tracks. Scheduled ingestion and metadata refresh are additional workflows. The graph follows README statements and shown source/index references; unsampled route details and refresh dispatch are not asserted beyond documented behavior.

Main entry points: [`scraper/app/main.py`](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/main.py) (FastAPI service), [`scraper/app/ingestion/pipeline.py`](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/pipeline.py) (Scrape pipeline), [`scraper/app/static/index.html`](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/static/index.html) (Web library), [`scraper/app/db/warehouse.py`](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/db/warehouse.py) (Track warehouse)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User experience"]
  node_frontend["Web library<br/>[index.html]"]
  node_catalog["Playlist catalog<br/>[playlists.py]"]
  node_soundcloud_player["SoundCloud player"]
end

subgraph group_api["API and jobs"]
  node_api_service["FastAPI service<br/>[main.py]"]
  node_job_tracker["Scrape job tracker<br/>[jobs.py]"]
  node_api_models["API data models<br/>[models.py]"]
  node_auth["Login and tokens<br/>[auth.py]"]
end

subgraph group_collection["Scraping and ingestion"]
  node_scrape_pipeline["Scrape pipeline<br/>[pipeline.py]"]
  node_scraper["SoundCloud scraper<br/>[soundcloud.py]"]
  node_scheduled_ingest["Scheduled ingestion<br/>[ingest.py]"]
  node_metadata_refresh["Metadata refresh"]
end

subgraph group_data["Persistence and identity"]
  node_lake[("Raw data lake<br/>[lake.py]")]
  node_warehouse[("Track warehouse<br/>[warehouse.py]")]
  node_postgres[("Postgres<br/>[init.sql]")]
end

node_user(("Listener / operator"))

node_user -->|"uses"| node_frontend
node_user -->|"submits scrape"| node_api_service
node_frontend -->|"calls API"| node_api_service
node_frontend -->|"shows catalog"| node_catalog
node_frontend -->|"embeds playback"| node_soundcloud_player
node_api_service -->|"tracks jobs"| node_job_tracker
node_api_service -->|"starts scrape"| node_scrape_pipeline
node_api_service -->|"queries library"| node_warehouse
node_api_service -->|"handles login"| node_auth
node_api_service -->|"serves playlists"| node_catalog
node_scrape_pipeline -->|"fetches and parses"| node_scraper
node_scrape_pipeline -->|"writes raw scrape"| node_lake
node_scrape_pipeline -->|"upserts tracks"| node_warehouse
node_scheduled_ingest -->|"runs pipeline"| node_scrape_pipeline
node_metadata_refresh -.->|"uses refresh flow"| node_scrape_pipeline
node_scrape_pipeline -->|"updates metadata"| node_warehouse
node_warehouse -->|"stores tracks"| node_postgres
node_auth -->|"issues token"| node_user

click node_frontend "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/static/index.html"
click node_catalog "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/scraping/playlists.py"
click node_api_service "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/main.py"
click node_job_tracker "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/jobs.py"
click node_api_models "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/models.py"
click node_auth "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/auth.py"
click node_scrape_pipeline "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/pipeline.py"
click node_scraper "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/scraping/soundcloud.py"
click node_scheduled_ingest "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/ingest.py"
click node_metadata_refresh "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/refresh_metadata.py"
click node_lake "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/lake.py"
click node_warehouse "https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/db/warehouse.py"
click node_postgres "https://github.com/kpdata-sci/apollo-audio/blob/main/warehouse/init.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_catalog,node_soundcloud_player toneBlue
class node_api_service,node_job_tracker,node_api_models,node_auth toneAmber
class node_scrape_pipeline,node_scraper,node_scheduled_ingest,node_metadata_refresh toneMint
class node_lake,node_warehouse,node_postgres toneRose
class node_user toneIndigo
```

## Components

### User experience

- [Web library](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/static/index.html): `scraper/app/static/index.html`
- [Playlist catalog](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/scraping/playlists.py): `scraper/app/scraping/playlists.py`
- **SoundCloud player**

### API and jobs

- [FastAPI service](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/main.py): `scraper/app/main.py`
- [Scrape job tracker](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/jobs.py): `scraper/app/jobs.py`
- [API data models](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/models.py): `scraper/app/models.py`
- [Login and tokens](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/auth.py): `scraper/app/auth.py`

### Scraping and ingestion

- [Scrape pipeline](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/pipeline.py): `scraper/app/ingestion/pipeline.py`
- [SoundCloud scraper](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/scraping/soundcloud.py): `scraper/app/scraping/soundcloud.py`
- [Scheduled ingestion](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/ingest.py): `scraper/app/ingestion/ingest.py`
- [Metadata refresh](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/refresh_metadata.py): `scraper/app/ingestion/refresh_metadata.py`

### Persistence and identity

- [Raw data lake](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/ingestion/lake.py): `scraper/app/ingestion/lake.py`
- [Track warehouse](https://github.com/kpdata-sci/apollo-audio/blob/main/scraper/app/db/warehouse.py): `scraper/app/db/warehouse.py`
- [Postgres](https://github.com/kpdata-sci/apollo-audio/blob/main/warehouse/init.sql): `warehouse/init.sql`

### Other components

- **Listener / operator**

## Connections

- Listener / operator → Web library: uses (evidence: `README.md`)
- Listener / operator → FastAPI service: submits scrape (evidence: `README.md`)
- Web library → FastAPI service: calls API (evidence: `README.md`)
- Web library → Playlist catalog: shows catalog (evidence: `README.md`)
- Web library → SoundCloud player: embeds playback (evidence: `README.md`)
- FastAPI service → Scrape job tracker: tracks jobs (evidence: `scraper/app/main.py`)
- FastAPI service → Scrape pipeline: starts scrape (evidence: `scraper/app/main.py`)
- FastAPI service → Track warehouse: queries library (evidence: `scraper/app/main.py`)
- FastAPI service → Login and tokens: handles login (evidence: `scraper/app/main.py`)
- FastAPI service → Playlist catalog: serves playlists (evidence: `README.md`)
- Scrape pipeline → SoundCloud scraper: fetches and parses (evidence: `scraper/app/ingestion/pipeline.py`)
- Scrape pipeline → Raw data lake: writes raw scrape (evidence: `scraper/app/ingestion/pipeline.py`)
- Scrape pipeline → Track warehouse: upserts tracks (evidence: `scraper/app/ingestion/pipeline.py`)
- Scheduled ingestion → Scrape pipeline: runs pipeline (evidence: `scraper/app/ingestion/ingest.py`)
- Metadata refresh → Scrape pipeline: uses refresh flow (evidence: `scraper/app/ingestion/refresh_metadata.py`)
- Scrape pipeline → Track warehouse: updates metadata (evidence: `scraper/app/ingestion/pipeline.py`)
- Track warehouse → Postgres: stores tracks (evidence: `scraper/app/db/warehouse.py`)
- Login and tokens → Listener / operator: issues token (evidence: `scraper/app/auth.py`)