# vaishnavibhojak1005/meteor architecture

> Architecture diagram of the GitHub repository vaishnavibhojak1005/meteor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaishnavibhojak1005/meteor
- Repository: https://github.com/vaishnavibhojak1005/meteor

## Overview

Meteor is a data-observability platform: data arrives through batch or streaming paths, is transformed and checked, and engineers inspect quality, anomalies, incidents, and lineage in a dashboard. The graph includes the documented AWS capabilities alongside the sampled local API and engine paths. The README describes SNS alerting and AWS integrations, but sampled source does not establish their dispatch wiring; those links are therefore omitted.

Main entry points: [`api/main.py`](https://github.com/vaishnavibhojak1005/meteor/blob/main/api/main.py) (FastAPI), [`incident_engine/incident_manager.py`](https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/incident_manager.py) (Incident Lifecycle), [`frontend/src/App.jsx`](https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/App.jsx) (Dashboard App), [`glue_jobs/orders_etl.py`](https://github.com/vaishnavibhojak1005/meteor/blob/main/glue_jobs/orders_etl.py) (Glue ETL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and ETL"]
  node_glue["Glue ETL<br/>[orders_etl.py]"]
  node_stream["Stream Producers"]
  node_localstream["Local Stream<br/>[local_consumer.py]"]
  node_lambda["S3 Quality Check<br/>[quality_check.py]"]
end

subgraph group_observe["Quality and Detection"]
  node_quality["Quality Checks<br/>[quality.py]"]
  node_profiler["Dataset Profiling<br/>[profiler.py]"]
  node_statistical["Statistical Anomalies<br/>[statistical.py]"]
  node_isolation["ML Anomalies"]
  node_baseline["Run Baselines<br/>[baseline.py]"]
end

subgraph group_incident["Incident Management"]
  node_incidentlogic["Severity and Causes<br/>[severity.py]"]
  node_rootcause["Root Cause Analysis<br/>[root_cause.py]"]
  node_incidentmgr["Incident Lifecycle"]
end

subgraph group_backend["API and State"]
  node_api["FastAPI<br/>[main.py]"]
  node_lineage["Lineage and Impact<br/>[lineage_manager.py]"]
end

subgraph group_frontend["Dashboard"]
  node_dashboard["Dashboard App<br/>[App.jsx]"]
  node_detail["Dataset Detail<br/>[DatasetDetail.jsx]"]
  node_incidentspage["Incidents View<br/>[Incidents.jsx]"]
end

node_operator(("Engineer"))
node_batch["CSV Data"]
node_s3[("Amazon S3")]
node_kinesis["Amazon Kinesis"]
node_postgres[("PostgreSQL")]
node_sns["Amazon SNS"]
node_athena["Amazon Athena<br/>[athena_query.sql]"]

node_batch -->|"uploads"| node_s3
node_s3 -->|"reads raw"| node_glue
node_glue -->|"writes Parquet"| node_s3
node_batch -->|"produces events"| node_stream
node_stream -->|"publishes"| node_kinesis
node_kinesis -.->|"streams events"| node_localstream
node_s3 -.->|"triggers upload check"| node_lambda
node_lambda -->|"reads CSV"| node_s3
node_operator -->|"views"| node_dashboard
node_dashboard -->|"routes to"| node_detail
node_dashboard -->|"routes to"| node_incidentspage
node_detail -->|"requests quality and anomalies"| node_api
node_incidentspage -->|"requests incidents"| node_api
node_api -->|"runs checks"| node_quality
node_api -->|"profiles datasets"| node_profiler
node_api -->|"detects anomalies"| node_statistical
node_api -->|"manages incidents"| node_incidentmgr
node_api -->|"computes impact"| node_lineage
node_incidentmgr -->|"classifies severity"| node_incidentlogic
node_incidentmgr -->|"analyzes cause"| node_rootcause
node_quality -.->|"stores quality state"| node_postgres
node_incidentmgr -->|"reads and writes incidents"| node_postgres
node_lineage -->|"reads dependencies"| node_postgres
node_api -->|"queries datasets"| node_postgres
node_baseline -.->|"loads run history"| node_postgres
node_isolation -->|"trains from history"| node_baseline
node_s3 -.->|"queries data"| node_athena

click node_glue "https://github.com/vaishnavibhojak1005/meteor/blob/main/glue_jobs/orders_etl.py"
click node_stream "https://github.com/vaishnavibhojak1005/meteor/blob/main/streaming/kinesis_producer.py"
click node_localstream "https://github.com/vaishnavibhojak1005/meteor/blob/main/streaming/local_consumer.py"
click node_lambda "https://github.com/vaishnavibhojak1005/meteor/blob/main/lambda_functions/quality_check.py"
click node_api "https://github.com/vaishnavibhojak1005/meteor/blob/main/api/main.py"
click node_quality "https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/quality.py"
click node_profiler "https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/profiler.py"
click node_statistical "https://github.com/vaishnavibhojak1005/meteor/blob/main/anomaly_engine/statistical.py"
click node_isolation "https://github.com/vaishnavibhojak1005/meteor/blob/main/anomaly_engine/isolation_forest.py"
click node_baseline "https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/baseline.py"
click node_incidentlogic "https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/severity.py"
click node_rootcause "https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/root_cause.py"
click node_incidentmgr "https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/incident_manager.py"
click node_lineage "https://github.com/vaishnavibhojak1005/meteor/blob/main/lineage/lineage_manager.py"
click node_dashboard "https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/App.jsx"
click node_detail "https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/pages/DatasetDetail.jsx"
click node_incidentspage "https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/pages/Incidents.jsx"
click node_athena "https://github.com/vaishnavibhojak1005/meteor/blob/main/athena_query.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_glue,node_stream,node_localstream,node_lambda toneBlue
class node_quality,node_profiler,node_statistical,node_isolation,node_baseline,node_s3,node_postgres toneAmber
class node_incidentlogic,node_rootcause,node_incidentmgr toneMint
class node_api,node_lineage toneRose
class node_dashboard,node_detail,node_incidentspage,node_operator,node_batch,node_kinesis,node_sns toneIndigo
class node_athena toneTeal
```

## Components

### Ingestion and ETL

- [Glue ETL](https://github.com/vaishnavibhojak1005/meteor/blob/main/glue_jobs/orders_etl.py): `glue_jobs/orders_etl.py`
- [Stream Producers](https://github.com/vaishnavibhojak1005/meteor/blob/main/streaming/kinesis_producer.py): `streaming/kinesis_producer.py`
- [Local Stream](https://github.com/vaishnavibhojak1005/meteor/blob/main/streaming/local_consumer.py): `streaming/local_consumer.py`
- [S3 Quality Check](https://github.com/vaishnavibhojak1005/meteor/blob/main/lambda_functions/quality_check.py): `lambda_functions/quality_check.py`

### Quality and Detection

- [Quality Checks](https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/quality.py): `quality_engine/quality.py`
- [Dataset Profiling](https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/profiler.py): `quality_engine/profiler.py`
- [Statistical Anomalies](https://github.com/vaishnavibhojak1005/meteor/blob/main/anomaly_engine/statistical.py): `anomaly_engine/statistical.py`
- [ML Anomalies](https://github.com/vaishnavibhojak1005/meteor/blob/main/anomaly_engine/isolation_forest.py): `anomaly_engine/isolation_forest.py`
- [Run Baselines](https://github.com/vaishnavibhojak1005/meteor/blob/main/quality_engine/baseline.py): `quality_engine/baseline.py`

### Incident Management

- [Severity and Causes](https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/severity.py): `incident_engine/severity.py`
- [Root Cause Analysis](https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/root_cause.py): `incident_engine/root_cause.py`
- [Incident Lifecycle](https://github.com/vaishnavibhojak1005/meteor/blob/main/incident_engine/incident_manager.py): `incident_engine/incident_manager.py`

### API and State

- [FastAPI](https://github.com/vaishnavibhojak1005/meteor/blob/main/api/main.py): `api/main.py`
- [Lineage and Impact](https://github.com/vaishnavibhojak1005/meteor/blob/main/lineage/lineage_manager.py): `lineage/lineage_manager.py`

### Dashboard

- [Dashboard App](https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Dataset Detail](https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/pages/DatasetDetail.jsx): `frontend/src/pages/DatasetDetail.jsx`
- [Incidents View](https://github.com/vaishnavibhojak1005/meteor/blob/main/frontend/src/pages/Incidents.jsx): `frontend/src/pages/Incidents.jsx`

### Other components

- **Engineer**
- **CSV Data**
- **Amazon S3**
- **Amazon Kinesis**
- **PostgreSQL**
- **Amazon SNS**
- [Amazon Athena](https://github.com/vaishnavibhojak1005/meteor/blob/main/athena_query.sql): `athena_query.sql`

## Connections

- CSV Data → Amazon S3: uploads
- Amazon S3 → Glue ETL: reads raw
- Glue ETL → Amazon S3: writes Parquet
- CSV Data → Stream Producers: produces events
- Stream Producers → Amazon Kinesis: publishes
- Amazon Kinesis → Local Stream: streams events
- Amazon S3 → S3 Quality Check: triggers upload check
- S3 Quality Check → Amazon S3: reads CSV
- Engineer → Dashboard App: views
- Dashboard App → Dataset Detail: routes to
- Dashboard App → Incidents View: routes to
- Dataset Detail → FastAPI: requests quality and anomalies
- Incidents View → FastAPI: requests incidents
- FastAPI → Quality Checks: runs checks
- FastAPI → Dataset Profiling: profiles datasets
- FastAPI → Statistical Anomalies: detects anomalies
- FastAPI → Incident Lifecycle: manages incidents
- FastAPI → Lineage and Impact: computes impact
- Incident Lifecycle → Severity and Causes: classifies severity
- Incident Lifecycle → Root Cause Analysis: analyzes cause
- Quality Checks → PostgreSQL: stores quality state
- Incident Lifecycle → PostgreSQL: reads and writes incidents
- Lineage and Impact → PostgreSQL: reads dependencies
- FastAPI → PostgreSQL: queries datasets
- Run Baselines → PostgreSQL: loads run history
- ML Anomalies → Run Baselines: trains from history
- Amazon S3 → Amazon Athena: queries data