# terk93/sensor-pulse architecture

> Architecture diagram of the GitHub repository terk93/sensor-pulse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/terk93/sensor-pulse
- Repository: https://github.com/terk93/sensor-pulse

## Overview

Sensor Pulse is an industrial sensor analytics and predictive-maintenance pipeline for NASA CMAPSS data. Its primary workflow ingests readings into Bronze, validates and derives Silver data, then builds Gold analytics and ML features; anomaly detection and Remaining Useful Life (RUL) prediction consume those outputs. A separate simulator demonstrates streaming ingestion into Bronze, while a dbt project provides SQL staging and marts. Notebook and dbt internals are described from the README; their source was not sampled, so links are limited to documented behavior.

Main entry points: [`sensor_pulse_pipeline.ipynb`](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse_pipeline.ipynb) (Spark pipeline), [`sensor_pulse/models/staging/stg_sensor_readings.sql`](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/staging/stg_sensor_readings.sql) (Staging model), [`ingestion/simulate_sensor_stream.py`](https://github.com/terk93/sensor-pulse/blob/main/ingestion/simulate_sensor_stream.py) (Stream simulator), [`ingestion/ingest_to_bronze.py`](https://github.com/terk93/sensor-pulse/blob/main/ingestion/ingest_to_bronze.py) (Bronze ingestor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_simulator["Stream simulator"]
  node_streamfile["Incoming JSONL"]
  node_ingestor["Bronze ingestor"]
end

subgraph group_lakehouse["Medallion pipeline"]
  node_pipeline["Spark pipeline"]
  node_bronze[("Bronze Parquet")]
  node_silver[("Silver readings")]
  node_gold[("Gold tables")]
end

subgraph group_analytics["Analytics and ML"]
  node_anomaly["Anomaly detection"]
  node_rul["RUL model<br/>[rul_model.ipynb]"]
end

subgraph group_dbt["dbt transformations"]
  node_staging["Staging model"]
  node_health["Engine health mart"]
  node_features["ML features mart"]
  node_drift["Sensor drift mart"]
end

node_cmapss["CMAPSS data"]
node_duckdb[("DuckDB")]

node_cmapss -->|"provides readings"| node_pipeline
node_pipeline -->|"writes raw"| node_bronze
node_pipeline -->|"cleans and flags"| node_silver
node_pipeline -->|"builds analytics"| node_gold
node_simulator -->|"replays rows"| node_cmapss
node_simulator -->|"appends JSONL"| node_streamfile
node_ingestor -->|"reads batches"| node_streamfile
node_ingestor -->|"writes Parquet"| node_bronze
node_silver -->|"supplies readings"| node_anomaly
node_gold -->|"supplies features"| node_rul
node_duckdb -->|"runs model"| node_staging
node_duckdb -->|"runs model"| node_health
node_duckdb -->|"runs model"| node_features
node_duckdb -->|"runs model"| node_drift
node_staging -->|"transforms readings"| node_health
node_staging -->|"transforms readings"| node_features
node_staging -->|"transforms readings"| node_drift

click node_pipeline "https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse_pipeline.ipynb"
click node_simulator "https://github.com/terk93/sensor-pulse/blob/main/ingestion/simulate_sensor_stream.py"
click node_ingestor "https://github.com/terk93/sensor-pulse/blob/main/ingestion/ingest_to_bronze.py"
click node_anomaly "https://github.com/terk93/sensor-pulse/blob/main/anomaly_detection.ipynb"
click node_rul "https://github.com/terk93/sensor-pulse/blob/main/rul_model.ipynb"
click node_staging "https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/staging/stg_sensor_readings.sql"
click node_health "https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_engine_health.sql"
click node_features "https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_ml_features.sql"
click node_drift "https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_sensor_drift.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simulator,node_streamfile,node_ingestor toneBlue
class node_pipeline,node_bronze,node_silver,node_gold,node_duckdb toneAmber
class node_anomaly,node_rul toneMint
class node_staging,node_health,node_features,node_drift toneRose
class node_cmapss toneIndigo
```

## Components

### Data ingestion

- [Stream simulator](https://github.com/terk93/sensor-pulse/blob/main/ingestion/simulate_sensor_stream.py): `ingestion/simulate_sensor_stream.py`
- **Incoming JSONL**
- [Bronze ingestor](https://github.com/terk93/sensor-pulse/blob/main/ingestion/ingest_to_bronze.py): `ingestion/ingest_to_bronze.py`

### Medallion pipeline

- [Spark pipeline](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse_pipeline.ipynb): `sensor_pulse_pipeline.ipynb`
- **Bronze Parquet**
- **Silver readings**
- **Gold tables**

### Analytics and ML

- [Anomaly detection](https://github.com/terk93/sensor-pulse/blob/main/anomaly_detection.ipynb): `anomaly_detection.ipynb`
- [RUL model](https://github.com/terk93/sensor-pulse/blob/main/rul_model.ipynb): `rul_model.ipynb`

### dbt transformations

- [Staging model](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/staging/stg_sensor_readings.sql): `sensor_pulse/models/staging/stg_sensor_readings.sql`
- [Engine health mart](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_engine_health.sql): `sensor_pulse/models/marts/mart_engine_health.sql`
- [ML features mart](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_ml_features.sql): `sensor_pulse/models/marts/mart_ml_features.sql`
- [Sensor drift mart](https://github.com/terk93/sensor-pulse/blob/main/sensor_pulse/models/marts/mart_sensor_drift.sql): `sensor_pulse/models/marts/mart_sensor_drift.sql`

### Other components

- **CMAPSS data**
- **DuckDB**

## Connections

- CMAPSS data → Spark pipeline: provides readings
- Spark pipeline → Bronze Parquet: writes raw
- Spark pipeline → Silver readings: cleans and flags
- Spark pipeline → Gold tables: builds analytics
- Stream simulator → CMAPSS data: replays rows
- Stream simulator → Incoming JSONL: appends JSONL
- Bronze ingestor → Incoming JSONL: reads batches
- Bronze ingestor → Bronze Parquet: writes Parquet
- Silver readings → Anomaly detection: supplies readings
- Gold tables → RUL model: supplies features
- DuckDB → Staging model: runs model
- DuckDB → Engine health mart: runs model
- DuckDB → ML features mart: runs model
- DuckDB → Sensor drift mart: runs model
- Staging model → Engine health mart: transforms readings
- Staging model → ML features mart: transforms readings
- Staging model → Sensor drift mart: transforms readings