How arsahadu/marketlens works
MarketLens is a scheduled stock-market data platform: Yahoo Finance data is ingested, stored in PostgreSQL, transformed into analytics marts, served through FastAPI, and presented in Streamlit. The dashboard includes market overview, company exploration, and a session-only portfolio analyzer. Edges reflect the README, indexed references, and shown calls; DAG and SQL internals are only partially sampled, so their detailed wiring is not inferred.
Main entry points: database.py PostgreSQL market data · api_client.py API client · branding.py Branding and logos · theme.py Theme and formatting
Components 21
Data pipeline
- Stock extraction
extractors/yfinance_extractor.py - Raw data loader
extractors/db_loader.py - Airflow pipeline
dags/marketlens_pipeline.py - Mart transformation
transformers/moving_avg.py
Market data storage
- Raw schema
sql/raw/create_raw_schema.sql - Daily mart SQL
sql/mart/transform_daily.sql - Monthly mart SQL
sql/mart/transform_monthly.sql - PostgreSQL market data
api/database.py
API serving
- FastAPI application
api/main.py - Stock endpoints
api/routes.py - Response schemas
api/schemas.py - API database access
api/database.py
Streamlit dashboard
- Dashboard home
dashboard/Home.py - Market overview
dashboard/pages/1_Market_Overview.py - Company explorer
dashboard/pages/2_Company_Explorer.py - Portfolio analyzer
dashboard/pages/3_Portfolio_Analyzer.py
Dashboard presentation
- API client
dashboard/utils/api_client.py - Theme and formatting
dashboard/utils/theme.py - Branding and logos
dashboard/utils/branding.py
Other components
- Yahoo Finance
- Clearbit logo service
Connections 27
- Yahoo Finance → Stock extraction: provides prices
README.md - Airflow pipeline → Stock extraction: orchestrates extraction
README.md - Airflow pipeline → Raw data loader: orchestrates loading
README.md - Airflow pipeline → Mart transformation: orchestrates transformation
README.md - Raw data loader → PostgreSQL market data: writes raw data
README.md - Mart transformation → PostgreSQL market data: creates marts
README.md - Raw schema → PostgreSQL market data: defines raw tables
- Daily mart SQL → PostgreSQL market data: transforms daily data
- Monthly mart SQL → PostgreSQL market data: transforms monthly data
- FastAPI application → Stock endpoints: registers routes
api/main.py - Stock endpoints → API database access: queries data
api/routes.py - API database access → PostgreSQL market data: opens connections
api/database.py - Stock endpoints → Response schemas: validates responses
api/routes.py - Dashboard home → API client: requests stock data
dashboard/Home.py - Market overview → API client: requests stock data
dashboard/pages/1_Market_Overview.py - Company explorer → API client: requests stock data
dashboard/pages/2_Company_Explorer.py - Portfolio analyzer → API client: requests stock data
dashboard/pages/3_Portfolio_Analyzer.py - API client → FastAPI application: calls REST API
README.md - Dashboard home → Theme and formatting: uses styling
dashboard/Home.py - Dashboard home → Branding and logos: uses branding
dashboard/Home.py - Market overview → Theme and formatting: uses styling
dashboard/pages/1_Market_Overview.py - Market overview → Branding and logos: uses branding
dashboard/pages/1_Market_Overview.py - Company explorer → Theme and formatting: uses styling
dashboard/pages/2_Company_Explorer.py - Company explorer → Branding and logos: uses branding
dashboard/pages/2_Company_Explorer.py - Portfolio analyzer → Theme and formatting: uses styling
dashboard/pages/3_Portfolio_Analyzer.py - Portfolio analyzer → Branding and logos: uses branding
dashboard/pages/3_Portfolio_Analyzer.py - Branding and logos → Clearbit logo service: loads company logos
dashboard/utils/branding.py