How arturo-ga/hackathon-factored works
This repository is a personal v1.4 data pipeline and analysis project for LATAM Bank hackathon data—not the team’s later “Expediente Vivo v2” product. Its main workflow synchronizes restricted CSVs from S3, converts them into bronze Parquet, builds typed silver and consumer-oriented gold tables in DuckDB, checks silver data quality, and can publish layers to BigQuery. Analysis scripts separately profile the dataset and investigate signals. Pipeline calls are supported by source excerpts; the README documents the data layers and analysis capabilities, but does not establish links from individual analysis scripts to pipeline stages.
Main entry points: run.py Pipeline runner · config.py Pipeline config · publish_bigquery.py BigQuery publisher · bq.py BigQuery utilities
Components 19
Data ingestion
- Pipeline runner
pipeline/run.py - S3 extraction
pipeline/extract_s3.py - Bronze conversion
pipeline/bronze.py - Pipeline config
pipeline/config.py
Data transformation
- DuckDB warehouse
pipeline/warehouse.py - Silver transforms
pipeline/silver.py - Gold products
pipeline/gold.py
Quality and publishing
- Quality checks
pipeline/quality.py - BigQuery publisher
pipeline/publish_bigquery.py - BigQuery utilities
pipeline/bq.py
Dataset analysis
- Table profiling
analysis/profile_tables.py - Signal analysis
analysis/signal_checks.py - Cross-signal analysis
analysis/deep_signal.py - Analysis database
analysis/build_analysis_db.py - Report sanitization
analysis/sanitize_docs.py - Analysis report
analysis/pass3_report.py
Other components
- Data analyst
- Organizer S3
- Google BigQuery
Connections 28
- Pipeline runner → S3 extraction: invokes
pipeline/run.py - Pipeline runner → Bronze conversion: invokes
pipeline/run.py - Pipeline runner → Silver transforms: invokes
pipeline/run.py - Pipeline runner → Quality checks: invokes
pipeline/run.py - Pipeline runner → Gold products: invokes
pipeline/run.py - Pipeline runner → BigQuery publisher: invokes
pipeline/run.py - S3 extraction → Organizer S3: syncs CSVs
pipeline/extract_s3.py - S3 extraction → Pipeline config: reads settings
pipeline/extract_s3.py - Bronze conversion → Pipeline config: reads settings
pipeline/bronze.py - Bronze conversion → BigQuery utilities: uploads optionally
pipeline/bronze.py - Silver transforms → DuckDB warehouse: builds and exports
pipeline/silver.py - Silver transforms → Pipeline config: reads settings
pipeline/silver.py - Gold products → DuckDB warehouse: builds and exports
pipeline/gold.py - Gold products → Pipeline config: reads settings
pipeline/gold.py - Quality checks → DuckDB warehouse: uses connection
pipeline/quality.py - Quality checks → Pipeline config: reads contracts
pipeline/quality.py - BigQuery publisher → BigQuery utilities: uses utilities
pipeline/publish_bigquery.py - BigQuery publisher → DuckDB warehouse: renders views
pipeline/publish_bigquery.py - BigQuery publisher → Pipeline config: reads settings
pipeline/publish_bigquery.py - BigQuery utilities → Pipeline config: reads settings
pipeline/bq.py - BigQuery publisher → Google BigQuery: publishes layers
pipeline/publish_bigquery.py - BigQuery utilities → Google BigQuery: loads Parquet
pipeline/bq.py - Data analyst → Table profiling: profiles data
- Data analyst → Signal analysis: checks signals
- Data analyst → Cross-signal analysis: investigates signals
- Data analyst → Analysis database: prepares analysis
- Data analyst → Analysis report: reviews findings
- Analysis report → Report sanitization: uses sanitizer
analysis/pass3_report.py