# cristopherfelime/apollo-data-pipeline architecture

> Architecture diagram of the GitHub repository cristopherfelime/apollo-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cristopherfelime/apollo-data-pipeline
- Repository: https://github.com/cristopherfelime/apollo-data-pipeline

## Overview

Apollo is an asynchronous data pipeline with two runtime paths: a producer orchestrates review, financial-news, and synthetic-transaction collection, validates and publishes events to Kafka; a consumer daemon reads Kafka batches, persists them to PostgreSQL, and commits offsets only after persistence succeeds. The README is marked TBA, so capabilities and wiring here are based on the supplied tree and sampled source; unsampled implementation details are not inferred.

Main entry points: [`apollo/main.py`](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/main.py) (Pipeline orchestrator), [`apollo/consumer_daemon.py`](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/consumer_daemon.py) (Consumer daemon), [`apollo/scrapers/play_store.py`](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/play_store.py) (Play Store scraper), [`apollo/scrapers/marketaux.py`](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/marketaux.py) (Marketaux scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_playstore["Play Store scraper<br/>[play_store.py]"]
  node_marketaux["Marketaux scraper<br/>[marketaux.py]"]
  node_transactions["Transaction generator<br/>[transactions.py]"]
  node_userpool["Persistent user pool<br/>[user_pool.txt]"]
  node_base["Scraper interface<br/>[base.py]"]
end

subgraph group_publishing["Event publishing"]
  node_main["Pipeline orchestrator<br/>[main.py]"]
  node_producer["Kafka producer<br/>[producer.py]"]
end

subgraph group_persistence["Kafka persistence"]
  node_daemon["Consumer daemon<br/>[consumer_daemon.py]"]
  node_consumer["Kafka consumer<br/>[consumer.py]"]
  node_persister["Postgres persister<br/>[service.py]"]
end

subgraph group_contracts["Data contracts"]
  node_schemas["Payload schemas<br/>[schemas.py]"]
end

node_operator(("Operator"))
node_playapi(("Google Play"))
node_newsapi(("Marketaux API"))
node_kafka["Kafka broker"]
node_postgres[("PostgreSQL")]

node_operator -->|"starts"| node_main
node_main -->|"runs scraper"| node_playstore
node_main -->|"runs scraper"| node_marketaux
node_main -->|"generates events"| node_transactions
node_playstore -->|"fetches reviews"| node_playapi
node_marketaux -->|"fetches news"| node_newsapi
node_transactions -->|"reads and writes"| node_userpool
node_playstore -->|"validates payloads"| node_schemas
node_marketaux -->|"validates payloads"| node_schemas
node_transactions -->|"creates payloads"| node_schemas
node_main -->|"publishes events"| node_producer
node_producer -->|"sends records"| node_kafka
node_daemon -->|"polls batches"| node_consumer
node_consumer -->|"reads and commits"| node_kafka
node_daemon -->|"parses and persists"| node_persister
node_persister -->|"writes batches"| node_postgres
node_daemon -->|"commits after success"| node_consumer

click node_main "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/main.py"
click node_playstore "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/play_store.py"
click node_marketaux "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/marketaux.py"
click node_transactions "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/transactions.py"
click node_userpool "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/data/user_pool.txt"
click node_base "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/base.py"
click node_schemas "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/schemas.py"
click node_producer "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/kafka/producer.py"
click node_daemon "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/consumer_daemon.py"
click node_consumer "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/kafka/consumer.py"
click node_persister "https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/database/service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_playstore,node_marketaux,node_transactions,node_userpool,node_base toneBlue
class node_main,node_producer,node_postgres toneAmber
class node_daemon,node_consumer,node_persister,node_newsapi toneMint
class node_schemas toneRose
class node_operator,node_playapi,node_kafka toneIndigo
```

## Components

### Data collection

- [Play Store scraper](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/play_store.py): `apollo/scrapers/play_store.py`
- [Marketaux scraper](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/marketaux.py): `apollo/scrapers/marketaux.py`
- [Transaction generator](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/transactions.py): `apollo/scrapers/transactions.py`
- [Persistent user pool](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/data/user_pool.txt): `apollo/scrapers/data/user_pool.txt`
- [Scraper interface](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/scrapers/base.py): `apollo/scrapers/base.py`

### Event publishing

- [Pipeline orchestrator](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/main.py): `apollo/main.py`
- [Kafka producer](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/kafka/producer.py): `apollo/kafka/producer.py`

### Kafka persistence

- [Consumer daemon](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/consumer_daemon.py): `apollo/consumer_daemon.py`
- [Kafka consumer](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/kafka/consumer.py): `apollo/kafka/consumer.py`
- [Postgres persister](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/database/service.py): `apollo/database/service.py`

### Data contracts

- [Payload schemas](https://github.com/cristopherfelime/apollo-data-pipeline/blob/main/apollo/schemas.py): `apollo/schemas.py`

### Other components

- **Operator**
- **Google Play**
- **Marketaux API**
- **Kafka broker**
- **PostgreSQL**

## Connections

- Operator → Pipeline orchestrator: starts
- Pipeline orchestrator → Play Store scraper: runs scraper
- Pipeline orchestrator → Marketaux scraper: runs scraper
- Pipeline orchestrator → Transaction generator: generates events
- Play Store scraper → Google Play: fetches reviews
- Marketaux scraper → Marketaux API: fetches news
- Transaction generator → Persistent user pool: reads and writes
- Play Store scraper → Payload schemas: validates payloads
- Marketaux scraper → Payload schemas: validates payloads
- Transaction generator → Payload schemas: creates payloads
- Pipeline orchestrator → Kafka producer: publishes events
- Kafka producer → Kafka broker: sends records
- Consumer daemon → Kafka consumer: polls batches
- Kafka consumer → Kafka broker: reads and commits
- Consumer daemon → Postgres persister: parses and persists
- Postgres persister → PostgreSQL: writes batches
- Consumer daemon → Kafka consumer: commits after success