# zerry101/pep-temporal-lakehouse architecture

> Architecture diagram of the GitHub repository zerry101/pep-temporal-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zerry101/pep-temporal-lakehouse
- Repository: https://github.com/zerry101/pep-temporal-lakehouse
- Video tour (about a minute): https://gitdiagram.com/zerry101/pep-temporal-lakehouse/video

## Overview

This Databricks lakehouse turns OpenSanctions snapshots into two kinds of answers: whether a politically exposed person is still in office, and who ultimately owns a company under sanctions rules. The principal workflow is snapshot-aware ingestion, Silver entity and relationship tables, then Gold tenure reporting and ownership analysis. Separate branches cover effective ownership, ultimate owners, entity-resolution work, and ML features. README architecture documents the broad stage transitions; sampled SQL confirms some Silver-to-Gold transformations. Several notebook implementations were not sampled, so their internal calls are not inferred.

Main entry points: [`bronze`](https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze) (Snapshot statements), [`silver/00_silver_ddl.py`](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/00_silver_ddl.py) (Silver tables), [`silver/14_silver_ownership.ipynb`](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/14_silver_ownership.ipynb) (Ownership links), [`gold/42_gold_dim_person.py`](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/42_gold_dim_person.py) (Gold dimensions)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Snapshot ingestion"]
  node_snapshot_check["Snapshot check"]
  node_snapshot_control[("Version control")]
  node_bronze_load["Bronze ingestion"]
  node_bronze_statements[("Snapshot statements")]
end

subgraph group_silver["Silver data"]
  node_silver_schema[("Silver tables<br/>[00_silver_ddl.py]")]
  node_person["Person records"]
  node_post["Position records<br/>[11_silver_post.py]"]
  node_referent["Entity references"]
  node_occupancy["Occupancy records"]
  node_legal_entity["Legal entities"]
  node_ownership_links["Ownership links"]
  node_integrity["Integrity checks"]
end

subgraph group_gold["Gold reporting"]
  node_gold_dimensions["Gold dimensions"]
  node_gold_fact["Tenure fact"]
  node_gold_features["ML features"]
end

subgraph group_ownership["Ownership analysis"]
  node_cascade["50% cascade"]
  node_cascade_checks["Cascade checks"]
  node_effective_ownership["Effective ownership"]
  node_ultimate_owners["Ultimate owners"]
end

subgraph group_resolution["Entity resolution"]
  node_entity_resolution["Entity resolution"]
end

node_opensanctions(("OpenSanctions"))

node_opensanctions -->|"publishes version"| node_snapshot_check
node_snapshot_check -->|"compares version"| node_snapshot_control
node_snapshot_check -->|"dispatches load"| node_bronze_load
node_bronze_load -->|"appends snapshot"| node_bronze_statements
node_bronze_statements -->|"transforms statements"| node_person
node_bronze_statements -->|"transforms statements"| node_post
node_bronze_statements -->|"resolves references"| node_referent
node_bronze_statements -->|"populates tables"| node_silver_schema
node_silver_schema -->|"defines table"| node_occupancy
node_silver_schema -->|"defines table"| node_legal_entity
node_silver_schema -->|"defines table"| node_ownership_links
node_person -->|"reads people"| node_gold_dimensions
node_occupancy -->|"reads occupancy"| node_gold_dimensions
node_post -->|"reads positions"| node_gold_dimensions
node_occupancy -->|"builds tenure fact"| node_gold_fact
node_gold_dimensions -->|"joins dimensions"| node_gold_fact
node_gold_fact -.->|"supports features"| node_gold_features
node_ownership_links -->|"traverses ownership"| node_cascade
node_cascade -->|"checks results"| node_cascade_checks
node_ownership_links -->|"enumerates paths"| node_effective_ownership
node_ownership_links -->|"finds top owners"| node_ultimate_owners
node_bronze_statements -->|"supports matching"| node_entity_resolution
node_entity_resolution -.->|"may improve inputs"| node_cascade
node_silver_schema -.->|"checks tables"| node_integrity

click node_snapshot_check "https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze"
click node_snapshot_control "https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze"
click node_bronze_load "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/bronze/bronze_ingestion.ipynb"
click node_bronze_statements "https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze"
click node_silver_schema "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/00_silver_ddl.py"
click node_person "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/10_silver_person.py"
click node_post "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/11_silver_post.py"
click node_referent "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/12_silver_referent.py"
click node_occupancy "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/20_silver_occupancy.ipynb"
click node_legal_entity "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/13_silver_legal_entity.ipynb"
click node_ownership_links "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/14_silver_ownership.ipynb"
click node_integrity "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/30_silver_integrity.ipynb"
click node_gold_dimensions "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/42_gold_dim_person.py"
click node_gold_fact "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/50_gold_fact_occupancy_period.py"
click node_gold_features "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/60_gold_ml_tenure_features.ipynb"
click node_cascade "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2_cascade.ipynb"
click node_cascade_checks "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2b_checks.ipynb"
click node_effective_ownership "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2b_effective_ownership.ipynb"
click node_ultimate_owners "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase3_entity_resolution.ipynb"
click node_entity_resolution "https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase3_entity_resolution.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_snapshot_check,node_snapshot_control,node_bronze_load,node_bronze_statements toneBlue
class node_silver_schema,node_person,node_post,node_referent,node_occupancy,node_legal_entity,node_ownership_links,node_integrity toneAmber
class node_gold_dimensions,node_gold_fact,node_gold_features toneMint
class node_cascade,node_cascade_checks,node_effective_ownership,node_ultimate_owners toneRose
class node_entity_resolution,node_opensanctions toneIndigo
```

## Components

### Snapshot ingestion

- [Snapshot check](https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze): `bronze`
- [Version control](https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze): `bronze`
- [Bronze ingestion](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/bronze/bronze_ingestion.ipynb): `bronze/bronze_ingestion.ipynb`
- [Snapshot statements](https://github.com/zerry101/pep-temporal-lakehouse/tree/main/bronze): `bronze`

### Silver data

- [Silver tables](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/00_silver_ddl.py): `silver/00_silver_ddl.py`
- [Person records](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/10_silver_person.py): `silver/10_silver_person.py`
- [Position records](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/11_silver_post.py): `silver/11_silver_post.py`
- [Entity references](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/12_silver_referent.py): `silver/12_silver_referent.py`
- [Occupancy records](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/20_silver_occupancy.ipynb): `silver/20_silver_occupancy.ipynb`
- [Legal entities](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/13_silver_legal_entity.ipynb): `silver/13_silver_legal_entity.ipynb`
- [Ownership links](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/14_silver_ownership.ipynb): `silver/14_silver_ownership.ipynb`
- [Integrity checks](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/silver/30_silver_integrity.ipynb): `silver/30_silver_integrity.ipynb`

### Gold reporting

- [Gold dimensions](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/42_gold_dim_person.py): `gold/42_gold_dim_person.py`
- [Tenure fact](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/50_gold_fact_occupancy_period.py): `gold/50_gold_fact_occupancy_period.py`
- [ML features](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/gold/60_gold_ml_tenure_features.ipynb): `gold/60_gold_ml_tenure_features.ipynb`

### Ownership analysis

- [50% cascade](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2_cascade.ipynb): `adhoc/goal2_phase2_cascade.ipynb`
- [Cascade checks](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2b_checks.ipynb): `adhoc/goal2_phase2b_checks.ipynb`
- [Effective ownership](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase2b_effective_ownership.ipynb): `adhoc/goal2_phase2b_effective_ownership.ipynb`
- [Ultimate owners](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase3_entity_resolution.ipynb): `adhoc/goal2_phase3_entity_resolution.ipynb`

### Entity resolution

- [Entity resolution](https://github.com/zerry101/pep-temporal-lakehouse/blob/main/adhoc/goal2_phase3_entity_resolution.ipynb): `adhoc/goal2_phase3_entity_resolution.ipynb`

### Other components

- **OpenSanctions**

## Connections

- OpenSanctions → Snapshot check: publishes version (evidence: `README.md`)
- Snapshot check → Version control: compares version (evidence: `README.md`)
- Snapshot check → Bronze ingestion: dispatches load (evidence: `README.md`)
- Bronze ingestion → Snapshot statements: appends snapshot (evidence: `README.md`)
- Snapshot statements → Person records: transforms statements (evidence: `silver/10_silver_person.py`)
- Snapshot statements → Position records: transforms statements (evidence: `silver/11_silver_post.py`)
- Snapshot statements → Entity references: resolves references (evidence: `silver/12_silver_referent.py`)
- Snapshot statements → Silver tables: populates tables (evidence: `README.md`)
- Silver tables → Occupancy records: defines table (evidence: `silver/00_silver_ddl.py`)
- Silver tables → Legal entities: defines table (evidence: `silver/00_silver_ddl.py`)
- Silver tables → Ownership links: defines table (evidence: `silver/00_silver_ddl.py`)
- Person records → Gold dimensions: reads people (evidence: `gold/42_gold_dim_person.py`)
- Occupancy records → Gold dimensions: reads occupancy (evidence: `gold/42_gold_dim_person.py`)
- Position records → Gold dimensions: reads positions
- Occupancy records → Tenure fact: builds tenure fact (evidence: `gold/50_gold_fact_occupancy_period.py`)
- Gold dimensions → Tenure fact: joins dimensions (evidence: `gold/50_gold_fact_occupancy_period.py`)
- Tenure fact → ML features: supports features (evidence: `README.md`)
- Ownership links → 50% cascade: traverses ownership (evidence: `README.md`)
- 50% cascade → Cascade checks: checks results (evidence: `README.md`)
- Ownership links → Effective ownership: enumerates paths (evidence: `README.md`)
- Ownership links → Ultimate owners: finds top owners (evidence: `README.md`)
- Snapshot statements → Entity resolution: supports matching (evidence: `README.md`)
- Entity resolution → 50% cascade: may improve inputs (evidence: `README.md`)
- Silver tables → Integrity checks: checks tables (evidence: `README.md`)
