# xososun/theta-data-engineering-project architecture

> Architecture diagram of the GitHub repository xososun/theta-data-engineering-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xososun/theta-data-engineering-project
- Repository: https://github.com/xososun/theta-data-engineering-project

## Overview

This repository is a cross-city road-crash data project: profiling and canonical-schema design are in place, and Chicago, NYC, and UK raw-data extraction is implemented. The main implemented workflow is source files or the NYC API → a source-specific extractor → traceable raw batches with manifests. Adapter mappings, staging/curated transformation, executable data-quality checks, database loading, and Airflow orchestration are documented or scaffolded but not yet built. Edges reflect excerpted calls and README statements; unsampled or unimplemented stages are shown without inferred runtime links.

Main entry points: [`src/extract/chicago_extractor.py`](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/chicago_extractor.py) (Chicago extractor), [`src/extract/uk_extractor.py`](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/uk_extractor.py) (UK extractor), [`src/extract/nyc_extractor.py`](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/nyc_extractor.py) (NYC extractor), [`src/utils/raw_writer.py`](https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/raw_writer.py) (Raw writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Source acquisition"]
  node_chicago_extract["Chicago extractor"]
  node_uk_extract["UK extractor<br/>[uk_extractor.py]"]
  node_nyc_extract["NYC extractor<br/>[nyc_extractor.py]"]
  node_raw_layer[("Raw batches")]
end

subgraph group_shared["Ingestion support"]
  node_batch_ids["Batch IDs<br/>[batch.py]"]
  node_adapter_loader["Adapter loader<br/>[config.py]"]
  node_raw_writer["Raw writer<br/>[raw_writer.py]"]
end

subgraph group_design["Schema and profiling"]
  node_chicago_adapter["Chicago adapter<br/>[chicago.yaml]"]
  node_nyc_adapter["NYC adapter<br/>[nyc.yaml]"]
  node_uk_adapter["UK adapter<br/>[uk.yaml]"]
  node_canonical_schema["Canonical schema"]
  node_profile["Chicago profiling<br/>[profile_crashes.py]"]
  node_postgres_schema[("Postgres schema<br/>[schema.sql]")]
  node_data_dictionary["Data dictionary<br/>[data_dictionary.md]"]
end

subgraph group_planned["Planned processing"]
  node_transform["Raw-to-curated transform"]
  node_validation["Data-quality checks"]
  node_airflow["Airflow DAGs"]
  node_staging[("Staging layer")]
  node_curated[("Curated layer")]
end

node_chicago_input["Chicago CSV"]
node_uk_input["UK annual CSVs"]
node_nyc_api(("NYC Socrata API"))

node_chicago_input -->|"provides CSV"| node_chicago_extract
node_uk_input -->|"provides files"| node_uk_extract
node_nyc_api -->|"serves pages"| node_nyc_extract
node_chicago_extract -->|"loads config"| node_adapter_loader
node_uk_extract -->|"loads config"| node_adapter_loader
node_nyc_extract -->|"loads config"| node_adapter_loader
node_chicago_extract -->|"creates batch ID"| node_batch_ids
node_uk_extract -->|"creates batch ID"| node_batch_ids
node_nyc_extract -->|"creates batch ID"| node_batch_ids
node_chicago_extract -->|"copies file; writes manifest"| node_raw_writer
node_uk_extract -->|"copies files; writes manifest"| node_raw_writer
node_nyc_extract -->|"writes pages; manifest"| node_raw_writer
node_raw_writer -->|"writes batches"| node_raw_layer
node_chicago_extract -->|"loads settings"| node_chicago_adapter
node_uk_extract -->|"loads settings"| node_uk_adapter
node_nyc_extract -->|"loads settings"| node_nyc_adapter
node_profile -->|"reads CSV"| node_chicago_input
node_transform -.->|"planned mapping target"| node_canonical_schema
node_transform -.->|"planned output"| node_staging
node_transform -.->|"planned output"| node_curated
node_validation -.->|"rules documented in schema"| node_postgres_schema
node_airflow -.->|"planned orchestration"| node_chicago_extract
node_airflow -.->|"planned orchestration"| node_nyc_extract
node_airflow -.->|"planned orchestration"| node_uk_extract

click node_chicago_extract "https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/chicago_extractor.py"
click node_uk_extract "https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/uk_extractor.py"
click node_nyc_extract "https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/nyc_extractor.py"
click node_raw_layer "https://github.com/xososun/theta-data-engineering-project/tree/main/data/raw"
click node_batch_ids "https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/batch.py"
click node_adapter_loader "https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/config.py"
click node_raw_writer "https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/raw_writer.py"
click node_chicago_adapter "https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/chicago.yaml"
click node_nyc_adapter "https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/nyc.yaml"
click node_uk_adapter "https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/uk.yaml"
click node_canonical_schema "https://github.com/xososun/theta-data-engineering-project/blob/main/config/canonical_schema.yaml"
click node_profile "https://github.com/xososun/theta-data-engineering-project/blob/main/src/profiling/profile_crashes.py"
click node_postgres_schema "https://github.com/xososun/theta-data-engineering-project/blob/main/sql/schema.sql"
click node_data_dictionary "https://github.com/xososun/theta-data-engineering-project/blob/main/docs/data_dictionary.md"
click node_transform "https://github.com/xososun/theta-data-engineering-project/tree/main/src/transform"
click node_validation "https://github.com/xososun/theta-data-engineering-project/tree/main/src/validation"
click node_airflow "https://github.com/xososun/theta-data-engineering-project/tree/main/dags"
click node_staging "https://github.com/xososun/theta-data-engineering-project/tree/main/data/staging"
click node_curated "https://github.com/xososun/theta-data-engineering-project/tree/main/data/curated"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chicago_extract,node_uk_extract,node_nyc_extract,node_raw_layer toneBlue
class node_batch_ids,node_adapter_loader,node_raw_writer toneAmber
class node_chicago_adapter,node_nyc_adapter,node_uk_adapter,node_canonical_schema,node_profile,node_postgres_schema,node_data_dictionary,node_nyc_api toneMint
class node_transform,node_validation,node_airflow,node_staging,node_curated toneRose
class node_chicago_input,node_uk_input toneIndigo
```

## Components

### Source acquisition

- [Chicago extractor](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/chicago_extractor.py): `src/extract/chicago_extractor.py`
- [UK extractor](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/uk_extractor.py): `src/extract/uk_extractor.py`
- [NYC extractor](https://github.com/xososun/theta-data-engineering-project/blob/main/src/extract/nyc_extractor.py): `src/extract/nyc_extractor.py`
- [Raw batches](https://github.com/xososun/theta-data-engineering-project/tree/main/data/raw): `data/raw`

### Ingestion support

- [Batch IDs](https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/batch.py): `src/utils/batch.py`
- [Adapter loader](https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/config.py): `src/utils/config.py`
- [Raw writer](https://github.com/xososun/theta-data-engineering-project/blob/main/src/utils/raw_writer.py): `src/utils/raw_writer.py`

### Schema and profiling

- [Chicago adapter](https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/chicago.yaml): `config/adapters/chicago.yaml`
- [NYC adapter](https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/nyc.yaml): `config/adapters/nyc.yaml`
- [UK adapter](https://github.com/xososun/theta-data-engineering-project/blob/main/config/adapters/uk.yaml): `config/adapters/uk.yaml`
- [Canonical schema](https://github.com/xososun/theta-data-engineering-project/blob/main/config/canonical_schema.yaml): `config/canonical_schema.yaml`
- [Chicago profiling](https://github.com/xososun/theta-data-engineering-project/blob/main/src/profiling/profile_crashes.py): `src/profiling/profile_crashes.py`
- [Postgres schema](https://github.com/xososun/theta-data-engineering-project/blob/main/sql/schema.sql): `sql/schema.sql`
- [Data dictionary](https://github.com/xososun/theta-data-engineering-project/blob/main/docs/data_dictionary.md): `docs/data_dictionary.md`

### Planned processing

- [Raw-to-curated transform](https://github.com/xososun/theta-data-engineering-project/tree/main/src/transform): `src/transform`
- [Data-quality checks](https://github.com/xososun/theta-data-engineering-project/tree/main/src/validation): `src/validation`
- [Airflow DAGs](https://github.com/xososun/theta-data-engineering-project/tree/main/dags): `dags`
- [Staging layer](https://github.com/xososun/theta-data-engineering-project/tree/main/data/staging): `data/staging`
- [Curated layer](https://github.com/xososun/theta-data-engineering-project/tree/main/data/curated): `data/curated`

### Other components

- **Chicago CSV**
- **UK annual CSVs**
- **NYC Socrata API**

## Connections

- Chicago CSV → Chicago extractor: provides CSV (evidence: `README.md`)
- UK annual CSVs → UK extractor: provides files (evidence: `README.md`)
- NYC Socrata API → NYC extractor: serves pages (evidence: `README.md`)
- Chicago extractor → Adapter loader: loads config (evidence: `src/extract/chicago_extractor.py`)
- UK extractor → Adapter loader: loads config (evidence: `src/extract/uk_extractor.py`)
- NYC extractor → Adapter loader: loads config (evidence: `src/extract/nyc_extractor.py`)
- Chicago extractor → Batch IDs: creates batch ID (evidence: `src/extract/chicago_extractor.py`)
- UK extractor → Batch IDs: creates batch ID (evidence: `src/extract/uk_extractor.py`)
- NYC extractor → Batch IDs: creates batch ID (evidence: `src/extract/nyc_extractor.py`)
- Chicago extractor → Raw writer: copies file; writes manifest (evidence: `src/extract/chicago_extractor.py`)
- UK extractor → Raw writer: copies files; writes manifest (evidence: `src/extract/uk_extractor.py`)
- NYC extractor → Raw writer: writes pages; manifest (evidence: `src/extract/nyc_extractor.py`)
- Raw writer → Raw batches: writes batches (evidence: `src/utils/raw_writer.py`)
- Chicago extractor → Chicago adapter: loads settings (evidence: `README.md`)
- UK extractor → UK adapter: loads settings (evidence: `README.md`)
- NYC extractor → NYC adapter: loads settings (evidence: `src/extract/nyc_extractor.py`)
- Chicago profiling → Chicago CSV: reads CSV (evidence: `src/profiling/profile_crashes.py`)
- Raw-to-curated transform → Canonical schema: planned mapping target (evidence: `README.md`)
- Raw-to-curated transform → Staging layer: planned output (evidence: `README.md`)
- Raw-to-curated transform → Curated layer: planned output (evidence: `README.md`)
- Data-quality checks → Postgres schema: rules documented in schema (evidence: `README.md`)
- Airflow DAGs → Chicago extractor: planned orchestration (evidence: `README.md`)
- Airflow DAGs → NYC extractor: planned orchestration (evidence: `README.md`)
- Airflow DAGs → UK extractor: planned orchestration (evidence: `README.md`)
