# yashvardhansingroha/etl_monitor architecture

> Architecture diagram of the GitHub repository yashvardhansingroha/etl_monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yashvardhansingroha/etl_monitor
- Repository: https://github.com/yashvardhansingroha/etl_monitor

## Overview

This is a batch financial-data ETL and quality-monitoring application for five US-listed companies. SEC Company Facts are mapped to annual canonical metrics; Yahoo prices are standardized and Nasdaq closes are reconciled against them. The loader stores results in SQLite, the quality engine records explainable checks and anomaly flags, and Streamlit presents dataset health, alerts, and incident details. The README documents the runtime order and dashboard behavior; sampled code confirms key transformations and database reads. Some intermediate-file and quality-engine details are documented but not fully excerpted.

Main entry points: [`data/financial_data.db`](https://github.com/yashvardhansingroha/etl_monitor/blob/main/data/financial_data.db) (Financial data store), [`src/database.py`](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/database.py) (SQLite loader), [`src/ingest_market.py`](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_market.py) (Yahoo OHLCV ingest), [`src/ingest_sec.py`](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_sec.py) (SEC annual mapping)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_sec_mapping["SEC annual mapping<br/>[ingest_sec.py]"]
  node_market_ingest["Yahoo OHLCV ingest<br/>[ingest_market.py]"]
  node_reconcile["Close reconciliation<br/>[ingest_market.py]"]
end

subgraph group_quality["Quality analysis"]
  node_dq_engine["Quality rules<br/>[quality.py]"]
end

subgraph group_storage["Storage"]
  node_loader["SQLite loader<br/>[database.py]"]
  node_sqlite[("Financial data store<br/>[financial_data.db]")]
end

subgraph group_dashboard["Dashboard"]
  node_health["Dataset health<br/>[app.py]"]
  node_alerts["Alerts explorer<br/>[app.py]"]
  node_incident["Incident drilldown<br/>[app.py]"]
end

node_sec(("SEC Company Facts"))
node_yahoo(("Yahoo Finance"))
node_nasdaq(("Nasdaq history"))
node_analyst(("Monitor user"))

node_sec -->|"provides facts"| node_sec_mapping
node_sec_mapping -->|"supplies annual metrics"| node_loader
node_yahoo -->|"provides prices"| node_market_ingest
node_market_ingest -->|"supplies OHLCV"| node_loader
node_nasdaq -->|"provides closes"| node_reconcile
node_market_ingest -->|"compares closes"| node_reconcile
node_reconcile -->|"supplies comparisons"| node_loader
node_loader -->|"loads tables"| node_sqlite
node_sqlite -->|"provides records"| node_dq_engine
node_dq_engine -->|"writes issues"| node_sqlite
node_sqlite -->|"queries metrics"| node_health
node_sqlite -->|"queries issues"| node_alerts
node_sqlite -->|"queries issue details"| node_incident
node_analyst -->|"views health"| node_health
node_analyst -->|"filters alerts"| node_alerts
node_analyst -->|"drills into issue"| node_incident

click node_sec_mapping "https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_sec.py"
click node_market_ingest "https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_market.py"
click node_reconcile "https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_market.py"
click node_loader "https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/database.py"
click node_sqlite "https://github.com/yashvardhansingroha/etl_monitor/blob/main/data/financial_data.db"
click node_dq_engine "https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/quality.py"
click node_health "https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py"
click node_alerts "https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py"
click node_incident "https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sec_mapping,node_market_ingest,node_reconcile,node_analyst toneBlue
class node_dq_engine toneAmber
class node_loader,node_sqlite toneMint
class node_health,node_alerts,node_incident toneRose
class node_sec,node_yahoo,node_nasdaq toneIndigo
```

## Components

### Data ingestion

- [SEC annual mapping](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_sec.py): `src/ingest_sec.py`
- [Yahoo OHLCV ingest](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_market.py): `src/ingest_market.py`
- [Close reconciliation](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/ingest_market.py): `src/ingest_market.py`

### Quality analysis

- [Quality rules](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/quality.py): `src/quality.py`

### Storage

- [SQLite loader](https://github.com/yashvardhansingroha/etl_monitor/blob/main/src/database.py): `src/database.py`
- [Financial data store](https://github.com/yashvardhansingroha/etl_monitor/blob/main/data/financial_data.db): `data/financial_data.db`

### Dashboard

- [Dataset health](https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py): `dashboard/app.py`
- [Alerts explorer](https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py): `dashboard/app.py`
- [Incident drilldown](https://github.com/yashvardhansingroha/etl_monitor/blob/main/dashboard/app.py): `dashboard/app.py`

### Other components

- **SEC Company Facts**
- **Yahoo Finance**
- **Nasdaq history**
- **Monitor user**

## Connections

- SEC Company Facts → SEC annual mapping: provides facts (evidence: `README.md`)
- SEC annual mapping → SQLite loader: supplies annual metrics (evidence: `README.md`)
- Yahoo Finance → Yahoo OHLCV ingest: provides prices (evidence: `README.md`)
- Yahoo OHLCV ingest → SQLite loader: supplies OHLCV (evidence: `README.md`)
- Nasdaq history → Close reconciliation: provides closes (evidence: `README.md`)
- Yahoo OHLCV ingest → Close reconciliation: compares closes (evidence: `README.md`)
- Close reconciliation → SQLite loader: supplies comparisons (evidence: `README.md`)
- SQLite loader → Financial data store: loads tables (evidence: `README.md`)
- Financial data store → Quality rules: provides records (evidence: `src/quality.py`)
- Quality rules → Financial data store: writes issues (evidence: `README.md`)
- Financial data store → Dataset health: queries metrics (evidence: `dashboard/app.py`)
- Financial data store → Alerts explorer: queries issues (evidence: `dashboard/app.py`)
- Financial data store → Incident drilldown: queries issue details (evidence: `dashboard/app.py`)
- Monitor user → Dataset health: views health (evidence: `README.md`)
- Monitor user → Alerts explorer: filters alerts (evidence: `README.md`)
- Monitor user → Incident drilldown: drills into issue (evidence: `README.md`)