How terk93/sensor-pulse works
Sensor Pulse is an industrial sensor analytics and predictive-maintenance pipeline for NASA CMAPSS data. Its primary workflow ingests readings into Bronze, validates and derives Silver data, then builds Gold analytics and ML features; anomaly detection and Remaining Useful Life (RUL) prediction consume those outputs. A separate simulator demonstrates streaming ingestion into Bronze, while a dbt project provides SQL staging and marts. Notebook and dbt internals are described from the README; their source was not sampled, so links are limited to documented behavior.
Main entry points: sensor_pulse_pipeline.ipynb Spark pipeline · stg_sensor_readings.sql Staging model · simulate_sensor_stream.py Stream simulator · ingest_to_bronze.py Bronze ingestor
Components 15
Data ingestion
- Stream simulator
ingestion/simulate_sensor_stream.py - Incoming JSONL
- Bronze ingestor
ingestion/ingest_to_bronze.py
Medallion pipeline
- Spark pipeline
sensor_pulse_pipeline.ipynb - Bronze Parquet
- Silver readings
- Gold tables
Analytics and ML
- Anomaly detection
anomaly_detection.ipynb - RUL model
rul_model.ipynb
dbt transformations
- Staging model
sensor_pulse/models/staging/stg_sensor_readings.sql - Engine health mart
sensor_pulse/models/marts/mart_engine_health.sql - ML features mart
sensor_pulse/models/marts/mart_ml_features.sql - Sensor drift mart
sensor_pulse/models/marts/mart_sensor_drift.sql
Other components
- CMAPSS data
- DuckDB
Connections 17
- CMAPSS data → Spark pipeline: provides readings
- Spark pipeline → Bronze Parquet: writes raw
- Spark pipeline → Silver readings: cleans and flags
- Spark pipeline → Gold tables: builds analytics
- Stream simulator → CMAPSS data: replays rows
- Stream simulator → Incoming JSONL: appends JSONL
- Bronze ingestor → Incoming JSONL: reads batches
- Bronze ingestor → Bronze Parquet: writes Parquet
- Silver readings → Anomaly detection: supplies readings
- Gold tables → RUL model: supplies features
- DuckDB → Staging model: runs model
- DuckDB → Engine health mart: runs model
- DuckDB → ML features mart: runs model
- DuckDB → Sensor drift mart: runs model
- Staging model → Engine health mart: transforms readings
- Staging model → ML features mart: transforms readings
- Staging model → Sensor drift mart: transforms readings