# xcaim04/la-ceiba-data architecture

> Architecture diagram of the GitHub repository xcaim04/la-ceiba-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xcaim04/la-ceiba-data
- Repository: https://github.com/xcaim04/la-ceiba-data

## Overview

La Ceiba is a synthetic-data analytics product for reconciling a wholesaler’s multi-currency sales and revealing true product margins. Its main workflow is raw sales, inventory, purchase, and exchange-rate files → bronze/silver/gold transformations → quality checks and reconciliation → BI and a private admin/panel. It also provides a BigQuery/Dataform equivalent. The README documents the domain stages; sampled code confirms local orchestration, transactional admin updates, and web dispatch. Some SQL/Dataform internals and BI wiring were not sampled, so those links follow documented architecture only.

Main entry points: [`pipeline.py`](https://github.com/xcaim04/la-ceiba-data/blob/main/pipeline.py) (Local pipeline), [`app/main.py`](https://github.com/xcaim04/la-ceiba-data/blob/main/app/main.py) (FastAPI site), [`sql/gold`](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/gold) (Gold reporting), [`sql/silver`](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/silver) (Silver modeling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data inputs"]
  node_rawfiles["Raw business files"]
  node_synthetic["Synthetic data generator<br/>[generar_datos.py]"]
  node_cloudload["BigQuery loader<br/>[cargar_bigquery.py]"]
end

subgraph group_local["Local analytics"]
  node_localpipe["Local pipeline<br/>[pipeline.py]"]
  node_bronzesql["Bronze ingestion"]
  node_silversql["Silver modeling"]
  node_goldsql["Gold reporting"]
  node_qualitysql["Quality checks"]
  node_duckdb[("DuckDB database<br/>[datos.py]")]
end

subgraph group_cloud["BigQuery analytics"]
  node_dataformbronze["Bronze declarations<br/>[fuentes.js]"]
  node_dataformsilver["Silver models"]
  node_dataformgold["Gold models"]
  node_dataformquality["Dataform assertions"]
  node_bigquery[("BigQuery")]
end

subgraph group_web["Private site"]
  node_webapp["FastAPI site<br/>[main.py]"]
  node_security["Session security<br/>[seguridad.py]"]
  node_datamanage["Data and catalog admin<br/>[datos.py]"]
  node_dashboard["Private dashboard<br/>[dashboard.py]"]
end

subgraph group_reporting["BI reporting"]
  node_evidence["Evidence BI"]
end

node_operator(("Business operator"))
node_webuser(("Administrator"))

node_synthetic -->|"generates"| node_rawfiles
node_rawfiles -->|"feeds"| node_localpipe
node_localpipe -->|"executes"| node_bronzesql
node_localpipe -->|"executes"| node_silversql
node_localpipe -->|"executes"| node_goldsql
node_localpipe -->|"runs checks"| node_qualitysql
node_bronzesql -->|"classifies rows"| node_silversql
node_silversql -->|"produces summaries"| node_goldsql
node_goldsql -->|"validates outputs"| node_qualitysql
node_localpipe -->|"builds tables"| node_duckdb
node_cloudload -->|"loads bronze"| node_bigquery
node_cloudload -->|"prepares files"| node_rawfiles
node_dataformbronze -->|"declares sources"| node_bigquery
node_dataformbronze -->|"feeds models"| node_dataformsilver
node_dataformsilver -->|"feeds models"| node_dataformgold
node_dataformgold -->|"checked by"| node_dataformquality
node_dataformquality -->|"asserts tables"| node_bigquery
node_operator -->|"uses site"| node_webapp
node_webapp -->|"checks access"| node_security
node_webapp -->|"handles admin data"| node_datamanage
node_datamanage -->|"recalculates"| node_localpipe
node_webapp -->|"renders panel"| node_dashboard
node_webapp -->|"reads data"| node_duckdb
node_operator -->|"views analytics"| node_evidence
node_goldsql -->|"exports Parquet"| node_evidence
node_bigquery -.->|"serves gold tables"| node_evidence

click node_synthetic "https://github.com/xcaim04/la-ceiba-data/blob/main/generar/generar_datos.py"
click node_cloudload "https://github.com/xcaim04/la-ceiba-data/blob/main/cargar_bigquery.py"
click node_localpipe "https://github.com/xcaim04/la-ceiba-data/blob/main/pipeline.py"
click node_bronzesql "https://github.com/xcaim04/la-ceiba-data/tree/main/sql/bronze"
click node_silversql "https://github.com/xcaim04/la-ceiba-data/tree/main/sql/silver"
click node_goldsql "https://github.com/xcaim04/la-ceiba-data/tree/main/sql/gold"
click node_qualitysql "https://github.com/xcaim04/la-ceiba-data/tree/main/sql/calidad"
click node_duckdb "https://github.com/xcaim04/la-ceiba-data/blob/main/app/datos.py"
click node_dataformbronze "https://github.com/xcaim04/la-ceiba-data/blob/main/dataform/definitions/bronze/fuentes.js"
click node_dataformsilver "https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/silver"
click node_dataformgold "https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/gold"
click node_dataformquality "https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/calidad"
click node_evidence "https://github.com/xcaim04/la-ceiba-data/tree/main/bi"
click node_webapp "https://github.com/xcaim04/la-ceiba-data/blob/main/app/main.py"
click node_security "https://github.com/xcaim04/la-ceiba-data/blob/main/app/seguridad.py"
click node_datamanage "https://github.com/xcaim04/la-ceiba-data/blob/main/app/datos.py"
click node_dashboard "https://github.com/xcaim04/la-ceiba-data/blob/main/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rawfiles,node_synthetic,node_cloudload toneBlue
class node_localpipe,node_bronzesql,node_silversql,node_goldsql,node_qualitysql,node_duckdb toneAmber
class node_dataformbronze,node_dataformsilver,node_dataformgold,node_dataformquality,node_bigquery toneMint
class node_webapp,node_security,node_datamanage,node_dashboard toneRose
class node_evidence,node_operator,node_webuser toneIndigo
```

## Components

### Data inputs

- **Raw business files**
- [Synthetic data generator](https://github.com/xcaim04/la-ceiba-data/blob/main/generar/generar_datos.py): `generar/generar_datos.py`
- [BigQuery loader](https://github.com/xcaim04/la-ceiba-data/blob/main/cargar_bigquery.py): `cargar_bigquery.py`

### Local analytics

- [Local pipeline](https://github.com/xcaim04/la-ceiba-data/blob/main/pipeline.py): `pipeline.py`
- [Bronze ingestion](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/bronze): `sql/bronze`
- [Silver modeling](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/silver): `sql/silver`
- [Gold reporting](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/gold): `sql/gold`
- [Quality checks](https://github.com/xcaim04/la-ceiba-data/tree/main/sql/calidad): `sql/calidad`
- [DuckDB database](https://github.com/xcaim04/la-ceiba-data/blob/main/app/datos.py): `app/datos.py`

### BigQuery analytics

- [Bronze declarations](https://github.com/xcaim04/la-ceiba-data/blob/main/dataform/definitions/bronze/fuentes.js): `dataform/definitions/bronze/fuentes.js`
- [Silver models](https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/silver): `dataform/definitions/silver`
- [Gold models](https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/gold): `dataform/definitions/gold`
- [Dataform assertions](https://github.com/xcaim04/la-ceiba-data/tree/main/dataform/definitions/calidad): `dataform/definitions/calidad`
- **BigQuery**

### Private site

- [FastAPI site](https://github.com/xcaim04/la-ceiba-data/blob/main/app/main.py): `app/main.py`
- [Session security](https://github.com/xcaim04/la-ceiba-data/blob/main/app/seguridad.py): `app/seguridad.py`
- [Data and catalog admin](https://github.com/xcaim04/la-ceiba-data/blob/main/app/datos.py): `app/datos.py`
- [Private dashboard](https://github.com/xcaim04/la-ceiba-data/blob/main/dashboard.py): `dashboard.py`

### BI reporting

- [Evidence BI](https://github.com/xcaim04/la-ceiba-data/tree/main/bi): `bi`

### Other components

- **Business operator**
- **Administrator**

## Connections

- Synthetic data generator → Raw business files: generates (evidence: `README.md`)
- Raw business files → Local pipeline: feeds (evidence: `README.md`)
- Local pipeline → Bronze ingestion: executes (evidence: `pipeline.py`)
- Local pipeline → Silver modeling: executes (evidence: `pipeline.py`)
- Local pipeline → Gold reporting: executes (evidence: `pipeline.py`)
- Local pipeline → Quality checks: runs checks (evidence: `pipeline.py`)
- Bronze ingestion → Silver modeling: classifies rows (evidence: `README.md`)
- Silver modeling → Gold reporting: produces summaries (evidence: `README.md`)
- Gold reporting → Quality checks: validates outputs (evidence: `README.md`)
- Local pipeline → DuckDB database: builds tables (evidence: `pipeline.py`)
- BigQuery loader → BigQuery: loads bronze (evidence: `cargar_bigquery.py`)
- BigQuery loader → Raw business files: prepares files (evidence: `cargar_bigquery.py`)
- Bronze declarations → BigQuery: declares sources (evidence: `dataform/definitions/bronze/fuentes.js`)
- Bronze declarations → Silver models: feeds models (evidence: `README.md`)
- Silver models → Gold models: feeds models (evidence: `README.md`)
- Gold models → Dataform assertions: checked by (evidence: `README.md`)
- Dataform assertions → BigQuery: asserts tables (evidence: `README.md`)
- Business operator → FastAPI site: uses site (evidence: `app/main.py`)
- FastAPI site → Session security: checks access (evidence: `app/main.py`)
- FastAPI site → Data and catalog admin: handles admin data (evidence: `app/main.py`)
- Data and catalog admin → Local pipeline: recalculates (evidence: `app/datos.py`)
- FastAPI site → Private dashboard: renders panel (evidence: `app/main.py`)
- FastAPI site → DuckDB database: reads data (evidence: `app/main.py`)
- Business operator → Evidence BI: views analytics (evidence: `README.md`)
- Gold reporting → Evidence BI: exports Parquet
- BigQuery → Evidence BI: serves gold tables (evidence: `README.md`)