# fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp architecture

> Architecture diagram of the GitHub repository fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp
- Repository: https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp

## Overview

O produto é um preditor Ballon d’Or híbrido: temporadas encerradas entram por batch e a temporada ativa por streaming; os dados convergem no BigQuery para features e predições, e o modelo exportado é servido por uma API. O README documenta os principais componentes batch e SQL; as ligações internas do streaming e da API também aparecem nos trechos de código. O crawler e alguns passos de batch são descritos, mas não foram inspecionados além do README e dos trechos disponíveis.

Main entry points: [`pipeline/streaming_pipeline.py`](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/pipeline/streaming_pipeline.py) (Pipeline Beam), [`sql/06_gold_hibrida.sql`](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/06_gold_hibrida.sql) (Features e ranking), [`job/run_job.py`](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/job/run_job.py) (Publicador agendado), [`notebooks_t1`](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/tree/main/notebooks_t1) (Silver batch)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Coleta e publicação"]
  node_crawler["Crawler FBref"]
  node_cache[("Snapshot CSV")]
  node_scheduler["Cloud Scheduler"]
  node_job["Publicador agendado<br/>[run_job.py]"]
  node_pubsub["Pub/Sub"]
  node_replay["Publicador replay<br/>[publisher.py]"]
end

subgraph group_streaming["Processamento streaming"]
  node_beam["Pipeline Beam"]
  node_rules["Regras comuns<br/>[preprocessing.py]"]
  node_raw[("Bronze streaming<br/>[01_tables.sql]")]
  node_silverstream[("Silver streaming<br/>[01_tables.sql]")]
  node_deadletter[("Dead-letter<br/>[01_tables.sql]")]
  node_streamviews[("Views streaming<br/>[02_views.sql]")]
end

subgraph group_analytics["Batch e analytics"]
  node_batch["Batch histórico"]
  node_batchsql["Carga batch 2025–26<br/>[05_batch_2526.sql]"]
  node_silverbatch[("Silver batch")]
  node_hybrid[("View híbrida<br/>[04_hybrid_view.sql]")]
  node_gold[("Features e ranking")]
  node_batchpred[("Predições batch<br/>[07_top30_2026.sql]")]
end

subgraph group_serving["Modelo e inferência"]
  node_export["Exportação modelo"]
  node_vertex[("Vertex AI Registry")]
  node_api["API REST<br/>[main.py]"]
end

node_client(("Cliente HTTP"))

node_crawler -->|"grava snapshot"| node_cache
node_scheduler -->|"dispara diariamente"| node_job
node_cache -->|"lê snapshot"| node_job
node_job -->|"publica eventos"| node_pubsub
node_replay -->|"publica eventos"| node_pubsub
node_pubsub -->|"entrega mensagens"| node_beam
node_beam -->|"aplica parse"| node_rules
node_beam -->|"grava eventos brutos"| node_raw
node_beam -->|"grava válidos"| node_silverstream
node_beam -->|"grava inválidos"| node_deadletter
node_silverstream -->|"alimenta views"| node_streamviews
node_batch -->|"produz Silver"| node_silverbatch
node_batchsql -->|"atualiza histórico"| node_silverbatch
node_streamviews -->|"integra streaming"| node_hybrid
node_silverbatch -->|"integra batch"| node_hybrid
node_hybrid -->|"alimenta procedure"| node_gold
node_gold -->|"gera rankings"| node_batchpred
node_gold -->|"exporta modelo"| node_export
node_export -->|"registra artefato"| node_vertex
node_vertex -->|"fornece modelo"| node_api
node_client -->|"envia requisição"| node_api

click node_crawler "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/producer/crawlerlimpezaalemacompleto_corrigido.py"
click node_job "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/job/run_job.py"
click node_replay "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/producer/publisher.py"
click node_beam "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/pipeline/streaming_pipeline.py"
click node_rules "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/ballon_common/preprocessing.py"
click node_raw "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql"
click node_silverstream "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql"
click node_deadletter "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql"
click node_streamviews "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/02_views.sql"
click node_batch "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/tree/main/notebooks_t1"
click node_batchsql "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/05_batch_2526.sql"
click node_silverbatch "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/tree/main/notebooks_t1"
click node_hybrid "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/04_hybrid_view.sql"
click node_gold "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/06_gold_hibrida.sql"
click node_batchpred "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/07_top30_2026.sql"
click node_export "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/08_export_modelo.sql"
click node_api "https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/api/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler,node_cache,node_scheduler,node_job,node_pubsub,node_replay,node_client toneBlue
class node_beam,node_rules,node_raw,node_silverstream,node_deadletter,node_streamviews toneAmber
class node_batch,node_batchsql,node_silverbatch,node_hybrid,node_gold,node_batchpred toneMint
class node_export,node_vertex,node_api toneRose
```

## Components

### Coleta e publicação

- [Crawler FBref](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/producer/crawlerlimpezaalemacompleto_corrigido.py): `producer/crawlerlimpezaalemacompleto_corrigido.py`
- **Snapshot CSV**
- **Cloud Scheduler**
- [Publicador agendado](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/job/run_job.py): `job/run_job.py`
- **Pub/Sub**
- [Publicador replay](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/producer/publisher.py): `producer/publisher.py`

### Processamento streaming

- [Pipeline Beam](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/pipeline/streaming_pipeline.py): `pipeline/streaming_pipeline.py`
- [Regras comuns](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/ballon_common/preprocessing.py): `ballon_common/preprocessing.py`
- [Bronze streaming](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql): `sql/01_tables.sql`
- [Silver streaming](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql): `sql/01_tables.sql`
- [Dead-letter](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/01_tables.sql): `sql/01_tables.sql`
- [Views streaming](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/02_views.sql): `sql/02_views.sql`

### Batch e analytics

- [Batch histórico](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/tree/main/notebooks_t1): `notebooks_t1`
- [Carga batch 2025–26](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/05_batch_2526.sql): `sql/05_batch_2526.sql`
- [Silver batch](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/tree/main/notebooks_t1): `notebooks_t1`
- [View híbrida](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/04_hybrid_view.sql): `sql/04_hybrid_view.sql`
- [Features e ranking](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/06_gold_hibrida.sql): `sql/06_gold_hibrida.sql`
- [Predições batch](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/07_top30_2026.sql): `sql/07_top30_2026.sql`

### Modelo e inferência

- [Exportação modelo](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/sql/08_export_modelo.sql): `sql/08_export_modelo.sql`
- **Vertex AI Registry**
- [API REST](https://github.com/fsegurado/pdm_ballon-dor-hybrid-pipeline-gcp/blob/main/api/main.py): `api/main.py`

### Other components

- **Cliente HTTP**

## Connections

- Crawler FBref → Snapshot CSV: grava snapshot (evidence: `README.md`)
- Cloud Scheduler → Publicador agendado: dispara diariamente (evidence: `README.md`)
- Snapshot CSV → Publicador agendado: lê snapshot (evidence: `job/run_job.py`)
- Publicador agendado → Pub/Sub: publica eventos (evidence: `job/run_job.py`)
- Publicador replay → Pub/Sub: publica eventos (evidence: `producer/publisher.py`)
- Pub/Sub → Pipeline Beam: entrega mensagens (evidence: `pipeline/streaming_pipeline.py`)
- Pipeline Beam → Regras comuns: aplica parse (evidence: `pipeline/streaming_pipeline.py`)
- Pipeline Beam → Bronze streaming: grava eventos brutos (evidence: `README.md`)
- Pipeline Beam → Silver streaming: grava válidos (evidence: `README.md`)
- Pipeline Beam → Dead-letter: grava inválidos (evidence: `README.md`)
- Silver streaming → Views streaming: alimenta views (evidence: `README.md`)
- Batch histórico → Silver batch: produz Silver (evidence: `README.md`)
- Carga batch 2025–26 → Silver batch: atualiza histórico (evidence: `README.md`)
- Views streaming → View híbrida: integra streaming (evidence: `README.md`)
- Silver batch → View híbrida: integra batch (evidence: `README.md`)
- View híbrida → Features e ranking: alimenta procedure (evidence: `README.md`)
- Features e ranking → Predições batch: gera rankings (evidence: `README.md`)
- Features e ranking → Exportação modelo: exporta modelo (evidence: `README.md`)
- Exportação modelo → Vertex AI Registry: registra artefato (evidence: `README.md`)
- Vertex AI Registry → API REST: fornece modelo (evidence: `api/main.py`)
- Cliente HTTP → API REST: envia requisição (evidence: `README.md`)
