How yashvardhansingroha/etl_monitor works
This is a batch financial-data ETL and quality-monitoring application for five US-listed companies. SEC Company Facts are mapped to annual canonical metrics; Yahoo prices are standardized and Nasdaq closes are reconciled against them. The loader stores results in SQLite, the quality engine records explainable checks and anomaly flags, and Streamlit presents dataset health, alerts, and incident details. The README documents the runtime order and dashboard behavior; sampled code confirms key transformations and database reads. Some intermediate-file and quality-engine details are documented but not fully excerpted.
Main entry points: financial_data.db Financial data store · database.py SQLite loader · ingest_market.py Yahoo OHLCV ingest · ingest_sec.py SEC annual mapping
Components 13
Data ingestion
- SEC annual mapping
src/ingest_sec.py - Yahoo OHLCV ingest
src/ingest_market.py - Close reconciliation
src/ingest_market.py
Quality analysis
- Quality rules
src/quality.py
Storage
- SQLite loader
src/database.py - Financial data store
data/financial_data.db
Dashboard
- Dataset health
dashboard/app.py - Alerts explorer
dashboard/app.py - Incident drilldown
dashboard/app.py
Other components
- SEC Company Facts
- Yahoo Finance
- Nasdaq history
- Monitor user
Connections 16
- SEC Company Facts → SEC annual mapping: provides facts
README.md - SEC annual mapping → SQLite loader: supplies annual metrics
README.md - Yahoo Finance → Yahoo OHLCV ingest: provides prices
README.md - Yahoo OHLCV ingest → SQLite loader: supplies OHLCV
README.md - Nasdaq history → Close reconciliation: provides closes
README.md - Yahoo OHLCV ingest → Close reconciliation: compares closes
README.md - Close reconciliation → SQLite loader: supplies comparisons
README.md - SQLite loader → Financial data store: loads tables
README.md - Financial data store → Quality rules: provides records
src/quality.py - Quality rules → Financial data store: writes issues
README.md - Financial data store → Dataset health: queries metrics
dashboard/app.py - Financial data store → Alerts explorer: queries issues
dashboard/app.py - Financial data store → Incident drilldown: queries issue details
dashboard/app.py - Monitor user → Dataset health: views health
README.md - Monitor user → Alerts explorer: filters alerts
README.md - Monitor user → Incident drilldown: drills into issue
README.md