How kpdata-sci/apollo-audio works
Apollo is a SoundCloud scraper and track library. Users discover curated playlists or submit playlist/profile URLs; the API queues scrape jobs, fetches and parses SoundCloud pages, then writes immutable raw JSON to the lake before upserting tracks into Postgres. The web UI browses, searches, plays and favorites stored tracks. Scheduled ingestion and metadata refresh are additional workflows. The graph follows README statements and shown source/index references; unsampled route details and refresh dispatch are not asserted beyond documented behavior.
Main entry points: main.py FastAPI service · pipeline.py Scrape pipeline · index.html Web library · warehouse.py Track warehouse
Components 15
User experience
- Web library
scraper/app/static/index.html - Playlist catalog
scraper/app/scraping/playlists.py - SoundCloud player
API and jobs
- FastAPI service
scraper/app/main.py - Scrape job tracker
scraper/app/jobs.py - API data models
scraper/app/models.py - Login and tokens
scraper/app/auth.py
Scraping and ingestion
- Scrape pipeline
scraper/app/ingestion/pipeline.py - SoundCloud scraper
scraper/app/scraping/soundcloud.py - Scheduled ingestion
scraper/app/ingestion/ingest.py - Metadata refresh
scraper/app/ingestion/refresh_metadata.py
Persistence and identity
- Raw data lake
scraper/app/ingestion/lake.py - Track warehouse
scraper/app/db/warehouse.py - Postgres
warehouse/init.sql
Other components
- Listener / operator
Connections 18
- Listener / operator → Web library: uses
README.md - Listener / operator → FastAPI service: submits scrape
README.md - Web library → FastAPI service: calls API
README.md - Web library → Playlist catalog: shows catalog
README.md - Web library → SoundCloud player: embeds playback
README.md - FastAPI service → Scrape job tracker: tracks jobs
scraper/app/main.py - FastAPI service → Scrape pipeline: starts scrape
scraper/app/main.py - FastAPI service → Track warehouse: queries library
scraper/app/main.py - FastAPI service → Login and tokens: handles login
scraper/app/main.py - FastAPI service → Playlist catalog: serves playlists
README.md - Scrape pipeline → SoundCloud scraper: fetches and parses
scraper/app/ingestion/pipeline.py - Scrape pipeline → Raw data lake: writes raw scrape
scraper/app/ingestion/pipeline.py - Scrape pipeline → Track warehouse: upserts tracks
scraper/app/ingestion/pipeline.py - Scheduled ingestion → Scrape pipeline: runs pipeline
scraper/app/ingestion/ingest.py - Metadata refresh → Scrape pipeline: uses refresh flow
scraper/app/ingestion/refresh_metadata.py - Scrape pipeline → Track warehouse: updates metadata
scraper/app/ingestion/pipeline.py - Track warehouse → Postgres: stores tracks
scraper/app/db/warehouse.py - Login and tokens → Listener / operator: issues token
scraper/app/auth.py