How cristopherfelime/apollo-data-pipeline works
Apollo is an asynchronous data pipeline with two runtime paths: a producer orchestrates review, financial-news, and synthetic-transaction collection, validates and publishes events to Kafka; a consumer daemon reads Kafka batches, persists them to PostgreSQL, and commits offsets only after persistence succeeds. The README is marked TBA, so capabilities and wiring here are based on the supplied tree and sampled source; unsampled implementation details are not inferred.
Main entry points: main.py Pipeline orchestrator · consumer_daemon.py Consumer daemon · play_store.py Play Store scraper · marketaux.py Marketaux scraper
Components 16
Data collection
- Play Store scraper
apollo/scrapers/play_store.py - Marketaux scraper
apollo/scrapers/marketaux.py - Transaction generator
apollo/scrapers/transactions.py - Persistent user pool
apollo/scrapers/data/user_pool.txt - Scraper interface
apollo/scrapers/base.py
Event publishing
- Pipeline orchestrator
apollo/main.py - Kafka producer
apollo/kafka/producer.py
Kafka persistence
- Consumer daemon
apollo/consumer_daemon.py - Kafka consumer
apollo/kafka/consumer.py - Postgres persister
apollo/database/service.py
Data contracts
- Payload schemas
apollo/schemas.py
Other components
- Operator
- Google Play
- Marketaux API
- Kafka broker
- PostgreSQL
Connections 17
- Operator → Pipeline orchestrator: starts
- Pipeline orchestrator → Play Store scraper: runs scraper
- Pipeline orchestrator → Marketaux scraper: runs scraper
- Pipeline orchestrator → Transaction generator: generates events
- Play Store scraper → Google Play: fetches reviews
- Marketaux scraper → Marketaux API: fetches news
- Transaction generator → Persistent user pool: reads and writes
- Play Store scraper → Payload schemas: validates payloads
- Marketaux scraper → Payload schemas: validates payloads
- Transaction generator → Payload schemas: creates payloads
- Pipeline orchestrator → Kafka producer: publishes events
- Kafka producer → Kafka broker: sends records
- Consumer daemon → Kafka consumer: polls batches
- Kafka consumer → Kafka broker: reads and commits
- Consumer daemon → Postgres persister: parses and persists
- Postgres persister → PostgreSQL: writes batches
- Consumer daemon → Kafka consumer: commits after success