How xososun/theta-data-engineering-project works
This repository is a cross-city road-crash data project: profiling and canonical-schema design are in place, and Chicago, NYC, and UK raw-data extraction is implemented. The main implemented workflow is source files or the NYC API → a source-specific extractor → traceable raw batches with manifests. Adapter mappings, staging/curated transformation, executable data-quality checks, database loading, and Airflow orchestration are documented or scaffolded but not yet built. Edges reflect excerpted calls and README statements; unsampled or unimplemented stages are shown without inferred runtime links.
Main entry points: chicago_extractor.py Chicago extractor · uk_extractor.py UK extractor · nyc_extractor.py NYC extractor · raw_writer.py Raw writer
Components 22
Source acquisition
- Chicago extractor
src/extract/chicago_extractor.py - UK extractor
src/extract/uk_extractor.py - NYC extractor
src/extract/nyc_extractor.py - Raw batches
data/raw
Ingestion support
- Batch IDs
src/utils/batch.py - Adapter loader
src/utils/config.py - Raw writer
src/utils/raw_writer.py
Schema and profiling
- Chicago adapter
config/adapters/chicago.yaml - NYC adapter
config/adapters/nyc.yaml - UK adapter
config/adapters/uk.yaml - Canonical schema
config/canonical_schema.yaml - Chicago profiling
src/profiling/profile_crashes.py - Postgres schema
sql/schema.sql - Data dictionary
docs/data_dictionary.md
Planned processing
- Raw-to-curated transform
src/transform - Data-quality checks
src/validation - Airflow DAGs
dags - Staging layer
data/staging - Curated layer
data/curated
Other components
- Chicago CSV
- UK annual CSVs
- NYC Socrata API
Connections 24
- Chicago CSV → Chicago extractor: provides CSV
README.md - UK annual CSVs → UK extractor: provides files
README.md - NYC Socrata API → NYC extractor: serves pages
README.md - Chicago extractor → Adapter loader: loads config
src/extract/chicago_extractor.py - UK extractor → Adapter loader: loads config
src/extract/uk_extractor.py - NYC extractor → Adapter loader: loads config
src/extract/nyc_extractor.py - Chicago extractor → Batch IDs: creates batch ID
src/extract/chicago_extractor.py - UK extractor → Batch IDs: creates batch ID
src/extract/uk_extractor.py - NYC extractor → Batch IDs: creates batch ID
src/extract/nyc_extractor.py - Chicago extractor → Raw writer: copies file; writes manifest
src/extract/chicago_extractor.py - UK extractor → Raw writer: copies files; writes manifest
src/extract/uk_extractor.py - NYC extractor → Raw writer: writes pages; manifest
src/extract/nyc_extractor.py - Raw writer → Raw batches: writes batches
src/utils/raw_writer.py - Chicago extractor → Chicago adapter: loads settings
README.md - UK extractor → UK adapter: loads settings
README.md - NYC extractor → NYC adapter: loads settings
src/extract/nyc_extractor.py - Chicago profiling → Chicago CSV: reads CSV
src/profiling/profile_crashes.py - Raw-to-curated transform → Canonical schema: planned mapping target
README.md - Raw-to-curated transform → Staging layer: planned output
README.md - Raw-to-curated transform → Curated layer: planned output
README.md - Data-quality checks → Postgres schema: rules documented in schema
README.md - Airflow DAGs → Chicago extractor: planned orchestration
README.md - Airflow DAGs → NYC extractor: planned orchestration
README.md - Airflow DAGs → UK extractor: planned orchestration
README.md