How shuaahvee/tap-spotify works
Project type and purpose
• This is a Singer “tap” implementation (tap-spotify) used to extract (“tap”) data from the Spotify Web API and emit it in the Singer JSON format.
• Built with the Meltano Tap SDK, it’s designed to be used standalone or orchestrated via Meltano in ETL/ELT pipelines.
File structure
• Top-level scripts/config:
• Makefile, tox.ini, pyproject.toml, meltano.yml (build, lint, test, orchestrator config)
• README.md (project description, usage)
• Python package “tap_spotify/”:
• auth.py: OAuth2 token acquisition (client_id, client_secret, refresh_token)
• client.py: Wrapper around Spotify HTTP endpoints
• pagination.py: Handles cursor‐style pagination for API calls
• schemas/: JSON schemas for each resource type (album, artist, track, etc.)
• streams.py: Defines Singer stream classes that use client + pagination + schemas
• tap.py: The Tap orchestration class (entry point, config validation, stream instantiation)
• tests/: Unit tests covering core functionality
README for additional insights
• Configuration options (client_id, client_secret, refresh_token)
• Authentication flow (using external web app to fetch a Spotify refresh token)
• Usage patterns (CLI flags, “discover” command, Meltano orchestration)
a. Main components
• CLI / Entry Point (tap-spotify binary)
• Tap Orchestrator (Tap class in tap.py)
• Auth Module (auth.py)
• HTTP Client (client.py)
• Pagination Engine (pagination.py)
• Stream Definitions (streams.py using JSON schemas)
• Schema Store (schemas/*.py files)
• Output / Singer Protocol (stdout JSON messages)
• External Spotify Web API
• (Optional) Meltano Orchestrator / Target plugin
b. Relationships & interactions
1. User invokes CLI → Tap Orchestrator
2. Tap reads config → Auth Module obtains access token
3. Tap instantiates Streams → each stream uses HTTP Client + Pagination
4. For each page: Client calls Spotify API → Pagination yields records
5. Stream applies JSON Schema → emits Singer format messages to stdout
6. Downstream: Meltano or any Singer target consumes stdout
c. Architectural patterns & principles
• Modular separation: auth, client, pagination, schemas, streams, tap
• Single Responsibility: each module does one job
• Pluggability: new streams or schemas can be added easily
• Config‐driven: credentials and scopes injected at runtime
d. Key technologies & libraries
• Python 3.x (pyproject.toml)
• Meltano Tap SDK (Singer SDK)
• Requests or an HTTP library (in client.py)
• JSON Schema validation
Elements
• Clear component labels (e.g., Auth Module, Pagination Engine, Spotify API)
• Annotations for protocols (OAuth2 token request, REST API calls, Singer JSON messages)