# arturo-ga/hackathon-factored architecture

> Architecture diagram of the GitHub repository arturo-ga/hackathon-factored, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arturo-ga/hackathon-factored
- Repository: https://github.com/arturo-ga/hackathon-factored

## Overview

This repository is a personal v1.4 data pipeline and analysis project for LATAM Bank hackathon data—not the team’s later “Expediente Vivo v2” product. Its main workflow synchronizes restricted CSVs from S3, converts them into bronze Parquet, builds typed silver and consumer-oriented gold tables in DuckDB, checks silver data quality, and can publish layers to BigQuery. Analysis scripts separately profile the dataset and investigate signals. Pipeline calls are supported by source excerpts; the README documents the data layers and analysis capabilities, but does not establish links from individual analysis scripts to pipeline stages.

Main entry points: [`pipeline/run.py`](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/run.py) (Pipeline runner), [`pipeline/config.py`](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/config.py) (Pipeline config), [`pipeline/publish_bigquery.py`](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/publish_bigquery.py) (BigQuery publisher), [`pipeline/bq.py`](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/bq.py) (BigQuery utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_run["Pipeline runner<br/>[run.py]"]
  node_extract["S3 extraction<br/>[extract_s3.py]"]
  node_bronze["Bronze conversion<br/>[bronze.py]"]
  node_config["Pipeline config<br/>[config.py]"]
end

subgraph group_transform["Data transformation"]
  node_warehouse[("DuckDB warehouse<br/>[warehouse.py]")]
  node_silver["Silver transforms<br/>[silver.py]"]
  node_gold["Gold products<br/>[gold.py]"]
end

subgraph group_validate["Quality and publishing"]
  node_quality["Quality checks<br/>[quality.py]"]
  node_publish["BigQuery publisher"]
  node_bq_utils["BigQuery utilities<br/>[bq.py]"]
end

subgraph group_analysis["Dataset analysis"]
  node_profiling["Table profiling<br/>[profile_tables.py]"]
  node_signal["Signal analysis<br/>[signal_checks.py]"]
  node_deep_signal["Cross-signal analysis<br/>[deep_signal.py]"]
  node_analysis_db["Analysis database"]
  node_sanitize["Report sanitization<br/>[sanitize_docs.py]"]
  node_report["Analysis report<br/>[pass3_report.py]"]
end

node_analyst(("Data analyst"))
node_s3["Organizer S3"]
node_bigquery["Google BigQuery"]

node_run -->|"invokes"| node_extract
node_run -->|"invokes"| node_bronze
node_run -->|"invokes"| node_silver
node_run -->|"invokes"| node_quality
node_run -->|"invokes"| node_gold
node_run -->|"invokes"| node_publish
node_extract -->|"syncs CSVs"| node_s3
node_extract -->|"reads settings"| node_config
node_bronze -->|"reads settings"| node_config
node_bronze -.->|"uploads optionally"| node_bq_utils
node_silver -->|"builds and exports"| node_warehouse
node_silver -->|"reads settings"| node_config
node_gold -->|"builds and exports"| node_warehouse
node_gold -->|"reads settings"| node_config
node_quality -->|"uses connection"| node_warehouse
node_quality -->|"reads contracts"| node_config
node_publish -->|"uses utilities"| node_bq_utils
node_publish -->|"renders views"| node_warehouse
node_publish -->|"reads settings"| node_config
node_bq_utils -->|"reads settings"| node_config
node_publish -->|"publishes layers"| node_bigquery
node_bq_utils -->|"loads Parquet"| node_bigquery
node_analyst -.->|"profiles data"| node_profiling
node_analyst -.->|"checks signals"| node_signal
node_analyst -.->|"investigates signals"| node_deep_signal
node_analyst -.->|"prepares analysis"| node_analysis_db
node_analyst -.->|"reviews findings"| node_report
node_report -->|"uses sanitizer"| node_sanitize

click node_run "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/run.py"
click node_extract "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/extract_s3.py"
click node_bronze "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/bronze.py"
click node_config "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/config.py"
click node_warehouse "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/warehouse.py"
click node_silver "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/silver.py"
click node_gold "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/gold.py"
click node_quality "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/quality.py"
click node_publish "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/publish_bigquery.py"
click node_bq_utils "https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/bq.py"
click node_profiling "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/profile_tables.py"
click node_signal "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/signal_checks.py"
click node_deep_signal "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/deep_signal.py"
click node_analysis_db "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/build_analysis_db.py"
click node_sanitize "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/sanitize_docs.py"
click node_report "https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/pass3_report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run,node_extract,node_bronze,node_config toneBlue
class node_warehouse,node_silver,node_gold toneAmber
class node_quality,node_publish,node_bq_utils toneMint
class node_profiling,node_signal,node_deep_signal,node_analysis_db,node_sanitize,node_report toneRose
class node_analyst,node_s3,node_bigquery toneIndigo
```

## Components

### Data ingestion

- [Pipeline runner](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/run.py): `pipeline/run.py`
- [S3 extraction](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/extract_s3.py): `pipeline/extract_s3.py`
- [Bronze conversion](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/bronze.py): `pipeline/bronze.py`
- [Pipeline config](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/config.py): `pipeline/config.py`

### Data transformation

- [DuckDB warehouse](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/warehouse.py): `pipeline/warehouse.py`
- [Silver transforms](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/silver.py): `pipeline/silver.py`
- [Gold products](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/gold.py): `pipeline/gold.py`

### Quality and publishing

- [Quality checks](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/quality.py): `pipeline/quality.py`
- [BigQuery publisher](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/publish_bigquery.py): `pipeline/publish_bigquery.py`
- [BigQuery utilities](https://github.com/arturo-ga/hackathon-factored/blob/main/pipeline/bq.py): `pipeline/bq.py`

### Dataset analysis

- [Table profiling](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/profile_tables.py): `analysis/profile_tables.py`
- [Signal analysis](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/signal_checks.py): `analysis/signal_checks.py`
- [Cross-signal analysis](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/deep_signal.py): `analysis/deep_signal.py`
- [Analysis database](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/build_analysis_db.py): `analysis/build_analysis_db.py`
- [Report sanitization](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/sanitize_docs.py): `analysis/sanitize_docs.py`
- [Analysis report](https://github.com/arturo-ga/hackathon-factored/blob/main/analysis/pass3_report.py): `analysis/pass3_report.py`

### Other components

- **Data analyst**
- **Organizer S3**
- **Google BigQuery**

## Connections

- Pipeline runner → S3 extraction: invokes (evidence: `pipeline/run.py`)
- Pipeline runner → Bronze conversion: invokes (evidence: `pipeline/run.py`)
- Pipeline runner → Silver transforms: invokes (evidence: `pipeline/run.py`)
- Pipeline runner → Quality checks: invokes (evidence: `pipeline/run.py`)
- Pipeline runner → Gold products: invokes (evidence: `pipeline/run.py`)
- Pipeline runner → BigQuery publisher: invokes (evidence: `pipeline/run.py`)
- S3 extraction → Organizer S3: syncs CSVs (evidence: `pipeline/extract_s3.py`)
- S3 extraction → Pipeline config: reads settings (evidence: `pipeline/extract_s3.py`)
- Bronze conversion → Pipeline config: reads settings (evidence: `pipeline/bronze.py`)
- Bronze conversion → BigQuery utilities: uploads optionally (evidence: `pipeline/bronze.py`)
- Silver transforms → DuckDB warehouse: builds and exports (evidence: `pipeline/silver.py`)
- Silver transforms → Pipeline config: reads settings (evidence: `pipeline/silver.py`)
- Gold products → DuckDB warehouse: builds and exports (evidence: `pipeline/gold.py`)
- Gold products → Pipeline config: reads settings (evidence: `pipeline/gold.py`)
- Quality checks → DuckDB warehouse: uses connection (evidence: `pipeline/quality.py`)
- Quality checks → Pipeline config: reads contracts (evidence: `pipeline/quality.py`)
- BigQuery publisher → BigQuery utilities: uses utilities (evidence: `pipeline/publish_bigquery.py`)
- BigQuery publisher → DuckDB warehouse: renders views (evidence: `pipeline/publish_bigquery.py`)
- BigQuery publisher → Pipeline config: reads settings (evidence: `pipeline/publish_bigquery.py`)
- BigQuery utilities → Pipeline config: reads settings (evidence: `pipeline/bq.py`)
- BigQuery publisher → Google BigQuery: publishes layers (evidence: `pipeline/publish_bigquery.py`)
- BigQuery utilities → Google BigQuery: loads Parquet (evidence: `pipeline/bq.py`)
- Data analyst → Table profiling: profiles data
- Data analyst → Signal analysis: checks signals
- Data analyst → Cross-signal analysis: investigates signals
- Data analyst → Analysis database: prepares analysis
- Data analyst → Analysis report: reviews findings
- Analysis report → Report sanitization: uses sanitizer (evidence: `analysis/pass3_report.py`)
