How vaishnavibhojak1005/meteor works
Meteor is a data-observability platform: data arrives through batch or streaming paths, is transformed and checked, and engineers inspect quality, anomalies, incidents, and lineage in a dashboard. The graph includes the documented AWS capabilities alongside the sampled local API and engine paths. The README describes SNS alerting and AWS integrations, but sampled source does not establish their dispatch wiring; those links are therefore omitted.
Main entry points: main.py FastAPI · incident_manager.py Incident Lifecycle · App.jsx Dashboard App · orders_etl.py Glue ETL
Components 24
Ingestion and ETL
- Glue ETL
glue_jobs/orders_etl.py - Stream Producers
streaming/kinesis_producer.py - Local Stream
streaming/local_consumer.py - S3 Quality Check
lambda_functions/quality_check.py
Quality and Detection
- Quality Checks
quality_engine/quality.py - Dataset Profiling
quality_engine/profiler.py - Statistical Anomalies
anomaly_engine/statistical.py - ML Anomalies
anomaly_engine/isolation_forest.py - Run Baselines
quality_engine/baseline.py
Incident Management
- Severity and Causes
incident_engine/severity.py - Root Cause Analysis
incident_engine/root_cause.py - Incident Lifecycle
incident_engine/incident_manager.py
API and State
- FastAPI
api/main.py - Lineage and Impact
lineage/lineage_manager.py
Dashboard
- Dashboard App
frontend/src/App.jsx - Dataset Detail
frontend/src/pages/DatasetDetail.jsx - Incidents View
frontend/src/pages/Incidents.jsx
Other components
- Engineer
- CSV Data
- Amazon S3
- Amazon Kinesis
- PostgreSQL
- Amazon SNS
- Amazon Athena
athena_query.sql
Connections 27
- CSV Data → Amazon S3: uploads
- Amazon S3 → Glue ETL: reads raw
- Glue ETL → Amazon S3: writes Parquet
- CSV Data → Stream Producers: produces events
- Stream Producers → Amazon Kinesis: publishes
- Amazon Kinesis → Local Stream: streams events
- Amazon S3 → S3 Quality Check: triggers upload check
- S3 Quality Check → Amazon S3: reads CSV
- Engineer → Dashboard App: views
- Dashboard App → Dataset Detail: routes to
- Dashboard App → Incidents View: routes to
- Dataset Detail → FastAPI: requests quality and anomalies
- Incidents View → FastAPI: requests incidents
- FastAPI → Quality Checks: runs checks
- FastAPI → Dataset Profiling: profiles datasets
- FastAPI → Statistical Anomalies: detects anomalies
- FastAPI → Incident Lifecycle: manages incidents
- FastAPI → Lineage and Impact: computes impact
- Incident Lifecycle → Severity and Causes: classifies severity
- Incident Lifecycle → Root Cause Analysis: analyzes cause
- Quality Checks → PostgreSQL: stores quality state
- Incident Lifecycle → PostgreSQL: reads and writes incidents
- Lineage and Impact → PostgreSQL: reads dependencies
- FastAPI → PostgreSQL: queries datasets
- Run Baselines → PostgreSQL: loads run history
- ML Anomalies → Run Baselines: trains from history
- Amazon S3 → Amazon Athena: queries data