# arsahadu/marketlens architecture

> Architecture diagram of the GitHub repository arsahadu/marketlens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arsahadu/marketlens
- Repository: https://github.com/arsahadu/marketlens

## Overview

MarketLens is a scheduled stock-market data platform: Yahoo Finance data is ingested, stored in PostgreSQL, transformed into analytics marts, served through FastAPI, and presented in Streamlit. The dashboard includes market overview, company exploration, and a session-only portfolio analyzer. Edges reflect the README, indexed references, and shown calls; DAG and SQL internals are only partially sampled, so their detailed wiring is not inferred.

Main entry points: [`api/database.py`](https://github.com/arsahadu/marketlens/blob/main/api/database.py) (PostgreSQL market data), [`dashboard/utils/api_client.py`](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/api_client.py) (API client), [`dashboard/utils/branding.py`](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/branding.py) (Branding and logos), [`dashboard/utils/theme.py`](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/theme.py) (Theme and formatting)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Data pipeline"]
  node_ingestion["Stock extraction"]
  node_loading["Raw data loader<br/>[db_loader.py]"]
  node_orchestration["Airflow pipeline"]
  node_transformer["Mart transformation<br/>[moving_avg.py]"]
end

subgraph group_storage["Market data storage"]
  node_rawschema["Raw schema"]
  node_dailytransform["Daily mart SQL"]
  node_monthlytransform["Monthly mart SQL"]
  node_postgres[("PostgreSQL market data<br/>[database.py]")]
end

subgraph group_serving["API serving"]
  node_apiapp["FastAPI application<br/>[main.py]"]
  node_routes["Stock endpoints<br/>[routes.py]"]
  node_schemas["Response schemas<br/>[schemas.py]"]
  node_apiaccess["API database access<br/>[database.py]"]
end

subgraph group_dashboard["Streamlit dashboard"]
  node_home["Dashboard home<br/>[Home.py]"]
  node_overview["Market overview"]
  node_explorer["Company explorer"]
  node_portfolio["Portfolio analyzer"]
end

subgraph group_presentation["Dashboard presentation"]
  node_client["API client<br/>[api_client.py]"]
  node_theme["Theme and formatting<br/>[theme.py]"]
  node_branding["Branding and logos<br/>[branding.py]"]
end

node_yahoo(("Yahoo Finance"))
node_logo_service(("Clearbit logo service"))

node_yahoo -->|"provides prices"| node_ingestion
node_orchestration -->|"orchestrates extraction"| node_ingestion
node_orchestration -->|"orchestrates loading"| node_loading
node_orchestration -->|"orchestrates transformation"| node_transformer
node_loading -->|"writes raw data"| node_postgres
node_transformer -->|"creates marts"| node_postgres
node_rawschema -.->|"defines raw tables"| node_postgres
node_dailytransform -.->|"transforms daily data"| node_postgres
node_monthlytransform -.->|"transforms monthly data"| node_postgres
node_apiapp -->|"registers routes"| node_routes
node_routes -->|"queries data"| node_apiaccess
node_apiaccess -->|"opens connections"| node_postgres
node_routes -->|"validates responses"| node_schemas
node_home -->|"requests stock data"| node_client
node_overview -->|"requests stock data"| node_client
node_explorer -->|"requests stock data"| node_client
node_portfolio -->|"requests stock data"| node_client
node_client -->|"calls REST API"| node_apiapp
node_home -->|"uses styling"| node_theme
node_home -->|"uses branding"| node_branding
node_overview -->|"uses styling"| node_theme
node_overview -->|"uses branding"| node_branding
node_explorer -->|"uses styling"| node_theme
node_explorer -->|"uses branding"| node_branding
node_portfolio -->|"uses styling"| node_theme
node_portfolio -->|"uses branding"| node_branding
node_branding -.->|"loads company logos"| node_logo_service

click node_ingestion "https://github.com/arsahadu/marketlens/blob/main/extractors/yfinance_extractor.py"
click node_loading "https://github.com/arsahadu/marketlens/blob/main/extractors/db_loader.py"
click node_orchestration "https://github.com/arsahadu/marketlens/blob/main/dags/marketlens_pipeline.py"
click node_transformer "https://github.com/arsahadu/marketlens/blob/main/transformers/moving_avg.py"
click node_rawschema "https://github.com/arsahadu/marketlens/blob/main/sql/raw/create_raw_schema.sql"
click node_dailytransform "https://github.com/arsahadu/marketlens/blob/main/sql/mart/transform_daily.sql"
click node_monthlytransform "https://github.com/arsahadu/marketlens/blob/main/sql/mart/transform_monthly.sql"
click node_postgres "https://github.com/arsahadu/marketlens/blob/main/api/database.py"
click node_apiapp "https://github.com/arsahadu/marketlens/blob/main/api/main.py"
click node_routes "https://github.com/arsahadu/marketlens/blob/main/api/routes.py"
click node_schemas "https://github.com/arsahadu/marketlens/blob/main/api/schemas.py"
click node_apiaccess "https://github.com/arsahadu/marketlens/blob/main/api/database.py"
click node_home "https://github.com/arsahadu/marketlens/blob/main/dashboard/Home.py"
click node_overview "https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/1_Market_Overview.py"
click node_explorer "https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/2_Company_Explorer.py"
click node_portfolio "https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/3_Portfolio_Analyzer.py"
click node_client "https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/api_client.py"
click node_theme "https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/theme.py"
click node_branding "https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/branding.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_loading,node_orchestration,node_transformer toneBlue
class node_rawschema,node_dailytransform,node_monthlytransform,node_postgres toneAmber
class node_apiapp,node_routes,node_schemas,node_apiaccess toneMint
class node_home,node_overview,node_explorer,node_portfolio toneRose
class node_client,node_theme,node_branding,node_yahoo,node_logo_service toneIndigo
```

## Components

### Data pipeline

- [Stock extraction](https://github.com/arsahadu/marketlens/blob/main/extractors/yfinance_extractor.py): `extractors/yfinance_extractor.py`
- [Raw data loader](https://github.com/arsahadu/marketlens/blob/main/extractors/db_loader.py): `extractors/db_loader.py`
- [Airflow pipeline](https://github.com/arsahadu/marketlens/blob/main/dags/marketlens_pipeline.py): `dags/marketlens_pipeline.py`
- [Mart transformation](https://github.com/arsahadu/marketlens/blob/main/transformers/moving_avg.py): `transformers/moving_avg.py`

### Market data storage

- [Raw schema](https://github.com/arsahadu/marketlens/blob/main/sql/raw/create_raw_schema.sql): `sql/raw/create_raw_schema.sql`
- [Daily mart SQL](https://github.com/arsahadu/marketlens/blob/main/sql/mart/transform_daily.sql): `sql/mart/transform_daily.sql`
- [Monthly mart SQL](https://github.com/arsahadu/marketlens/blob/main/sql/mart/transform_monthly.sql): `sql/mart/transform_monthly.sql`
- [PostgreSQL market data](https://github.com/arsahadu/marketlens/blob/main/api/database.py): `api/database.py`

### API serving

- [FastAPI application](https://github.com/arsahadu/marketlens/blob/main/api/main.py): `api/main.py`
- [Stock endpoints](https://github.com/arsahadu/marketlens/blob/main/api/routes.py): `api/routes.py`
- [Response schemas](https://github.com/arsahadu/marketlens/blob/main/api/schemas.py): `api/schemas.py`
- [API database access](https://github.com/arsahadu/marketlens/blob/main/api/database.py): `api/database.py`

### Streamlit dashboard

- [Dashboard home](https://github.com/arsahadu/marketlens/blob/main/dashboard/Home.py): `dashboard/Home.py`
- [Market overview](https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/1_Market_Overview.py): `dashboard/pages/1_Market_Overview.py`
- [Company explorer](https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/2_Company_Explorer.py): `dashboard/pages/2_Company_Explorer.py`
- [Portfolio analyzer](https://github.com/arsahadu/marketlens/blob/main/dashboard/pages/3_Portfolio_Analyzer.py): `dashboard/pages/3_Portfolio_Analyzer.py`

### Dashboard presentation

- [API client](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/api_client.py): `dashboard/utils/api_client.py`
- [Theme and formatting](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/theme.py): `dashboard/utils/theme.py`
- [Branding and logos](https://github.com/arsahadu/marketlens/blob/main/dashboard/utils/branding.py): `dashboard/utils/branding.py`

### Other components

- **Yahoo Finance**
- **Clearbit logo service**

## Connections

- Yahoo Finance → Stock extraction: provides prices (evidence: `README.md`)
- Airflow pipeline → Stock extraction: orchestrates extraction (evidence: `README.md`)
- Airflow pipeline → Raw data loader: orchestrates loading (evidence: `README.md`)
- Airflow pipeline → Mart transformation: orchestrates transformation (evidence: `README.md`)
- Raw data loader → PostgreSQL market data: writes raw data (evidence: `README.md`)
- Mart transformation → PostgreSQL market data: creates marts (evidence: `README.md`)
- Raw schema → PostgreSQL market data: defines raw tables
- Daily mart SQL → PostgreSQL market data: transforms daily data
- Monthly mart SQL → PostgreSQL market data: transforms monthly data
- FastAPI application → Stock endpoints: registers routes (evidence: `api/main.py`)
- Stock endpoints → API database access: queries data (evidence: `api/routes.py`)
- API database access → PostgreSQL market data: opens connections (evidence: `api/database.py`)
- Stock endpoints → Response schemas: validates responses (evidence: `api/routes.py`)
- Dashboard home → API client: requests stock data (evidence: `dashboard/Home.py`)
- Market overview → API client: requests stock data (evidence: `dashboard/pages/1_Market_Overview.py`)
- Company explorer → API client: requests stock data (evidence: `dashboard/pages/2_Company_Explorer.py`)
- Portfolio analyzer → API client: requests stock data (evidence: `dashboard/pages/3_Portfolio_Analyzer.py`)
- API client → FastAPI application: calls REST API (evidence: `README.md`)
- Dashboard home → Theme and formatting: uses styling (evidence: `dashboard/Home.py`)
- Dashboard home → Branding and logos: uses branding (evidence: `dashboard/Home.py`)
- Market overview → Theme and formatting: uses styling (evidence: `dashboard/pages/1_Market_Overview.py`)
- Market overview → Branding and logos: uses branding (evidence: `dashboard/pages/1_Market_Overview.py`)
- Company explorer → Theme and formatting: uses styling (evidence: `dashboard/pages/2_Company_Explorer.py`)
- Company explorer → Branding and logos: uses branding (evidence: `dashboard/pages/2_Company_Explorer.py`)
- Portfolio analyzer → Theme and formatting: uses styling (evidence: `dashboard/pages/3_Portfolio_Analyzer.py`)
- Portfolio analyzer → Branding and logos: uses branding (evidence: `dashboard/pages/3_Portfolio_Analyzer.py`)
- Branding and logos → Clearbit logo service: loads company logos (evidence: `dashboard/utils/branding.py`)
