# marketcalls/historify architecture

> Architecture diagram of the GitHub repository marketcalls/historify, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marketcalls/historify
- Repository: https://github.com/marketcalls/historify

## Overview

##### Project Type and Purpose
- This is a full-stack web application (“Historify”) for managing, visualizing, and scheduling historical stock market data downloads.
- Backend: Flask + SQLAlchemy + SQLite + APScheduler
- Frontend: Jinja2 templates + Tailwind CSS/DaisyUI + Vanilla ES6 JavaScript + TradingView Charts

##### File Structure Analysis
- Top-level:
- run.py – application entry point
- requirements.txt – dependencies
- README.md, docs/prd.md – high-level docs
- historify/app/:
- models/ – SQLAlchemy models & dynamic table definitions
- routes/ – Flask view functions (API endpoints + page routes)
- utils/ – data_fetcher, rate_limiter, scheduler helpers
- static/ – CSS, JS, images (design system, TradingView integration)
- templates/ – Jinja2 pages (dashboard, import/export, charts, scheduler)
- historify/.env.sample – environment configuration (API keys, DB URI, secrets)

##### Key Architectural Components
a. Frontend Layer
- Browser UI (HTML templates + CSS design system + JS modules)
- TradingView Lightweight Charts loaded client-side
- Command Palette, modals, tables, forms for import/export/watchlist/scheduler
b. Backend Layer
- Flask Application
– Routes split into API (JSON) vs. Page renderers
– Blueprints in app/routes/api.py, /charts.py, /scheduler.py, etc.
- Utils
– DataFetcher (bulk download, parallel threads, rate-limit)
– Scheduler (APScheduler jobs, IST timezone handling)
- Models
– Dynamic tables per symbol/interval
– Checkpoint, Watchlist, SchedulerJob, Settings
- Task Queue / Background Processing
– APScheduler for scheduled jobs
– ThreadPoolExecutor for parallel downloads
c. Database
- SQLite (DATABASE_URI from .env)
- SQLAlchemy ORM + dynamic table creation for efficient time-series storage
d. External Services
- OpenAlgo API (market data)
- TradingView JS library (client-side chart rendering)

##### Relationships and Data Flows
- User interacts with Browser UI → JS calls REST API endpoints (e.g., /api/download, /api/import-symbols)
- Flask routes handle requests → use utils/DataFetcher to call OpenAlgo API → write/read to SQLite via SQLAlchemy models
- Scheduler subsystem schedules jobs via APScheduler → triggers DataFetcher tasks in background → updates database
- Export requests queue CSV generation → streams result to user, monitors status via /api/export/queue
- Real-time watchlist/quotes: JS polling / GET /api/quotes → Flask returns JSON from models or external API

##### Architectural Patterns and Principles
- Separation of concerns: UI templates vs. API endpoints vs. data/model layer vs. scheduler services
- Modular routing using Flask Blueprints
- Dynamic table creation for scale (one table per symbol/interval)
- Background job scheduling (APScheduler) and parallel processing (ThreadPool)
- Rate limiting at utility level to respect API quotas
- Lazy loading / virtual scrolling for performance in UI

##### Essential Elements to Capture
- Frontend Modules: CSS design system, JS (dashboard.js, import.js, watchlist.js, scheduler.js, charts.js)
- Flask Blueprints: API vs. Page routes
- Utils: DataFetcher, RateLimiter, Scheduler helpers
- Scheduler Manager: APScheduler jobs configured in utils/scheduler.py
- Dynamic Data Model: Symbol/interval tables, checkpoint model
- Export Queue: background CSV streaming
- External Integrations: OpenAlgo (data), TradingView (charts)

##### Final Recommendations
- Don’t over-split: group similar utilities together (e.g., all data-fetching logic)
- Asynchronous/background flows separately from synchronous HTTP flows

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Browser"]:::actor

    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        Templates["Jinja2 Templates"]:::frontend
        CSS["CSS Design System"]:::frontend
        JS["JS Modules"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        direction TB
        subgraph "Flask Core"
            direction TB
            Init["App Init"]:::backend
            MainRoutes["Main/Page Routes"]:::backend
            APIRoutes["API Endpoints"]:::backend
            ChartRoutes["Charts Routes"]:::backend
            SchedulerRoutes["Scheduler Routes"]:::backend
            SettingsRoutes["Settings Routes"]:::backend
            WatchlistRoutes["Watchlist Routes"]:::backend
        end
        subgraph "Models"
            direction TB
            Checkpoint["Checkpoint Model"]:::model
            DynamicTables["Dynamic Tables"]:::model
            SchedulerJob["SchedulerJob Model"]:::model
            SettingsModel["Settings Model"]:::model
            StockData["StockData Model"]:::model
            WatchlistModel["Watchlist Model"]:::model
        end
        subgraph "Utils"
            direction TB
            Auth["Auth Helpers"]:::util
            DataFetcher["DataFetcher Service"]:::util
            DataFetcherChunked["Chunked DataFetcher"]:::util
            RateLimiter["Rate Limiter"]:::util
            SchedulerUtil["Scheduler Helpers"]:::util
        end
    end

    %% Data Layer
    DB["SQLite Database"]:::db

    %% Scheduler Subsystem
    SchedulerSub["Scheduler (APScheduler)"]:::scheduler

    %% External Services
    CloudOA["OpenAlgo API"]:::external
    CloudTV["TradingView CDN"]:::external

    %% Connections
    User -->|views pages| Templates
    User -->|loads styles| CSS
    User -->|runs scripts| JS

    JS -->|calls REST API| APIRoutes
    JS -->|loads charts lib| CloudTV

    MainRoutes -->|renders templates| Templates
    MainRoutes --> Models

    APIRoutes --> Auth
    APIRoutes --> DataFetcher
    APIRoutes --> DataFetcherChunked
    APIRoutes --> RateLimiter
    APIRoutes --> Models

    ChartRoutes --> Models

    SchedulerRoutes --> SchedulerUtil
    SchedulerRoutes --> Models

    SettingsRoutes --> Models
    WatchlistRoutes --> Models

    DataFetcher -->|HTTP requests| CloudOA
    DataFetcherChunked -->|HTTP requests| CloudOA
    DataFetcher -->|writes/reads| DB
    DataFetcherChunked -->|writes/reads| DB

    SchedulerSub --> SchedulerUtil
    SchedulerUtil --> DataFetcher
    SchedulerUtil --> DataFetcherChunked
    SchedulerUtil -->|schedules jobs| SchedulerSub
    SchedulerUtil --> DB

    Models -->|ORM read/write| DB

    %% Click Events
    click run "https://github.com/marketcalls/historify/blob/master/historify/run.py"
    click env "https://github.com/marketcalls/historify/blob/master/historify/.env.sample"
    click Init "https://github.com/marketcalls/historify/blob/master/historify/app/__init__.py"
    click Checkpoint "https://github.com/marketcalls/historify/blob/master/historify/app/models/checkpoint.py"
    click DynamicTables "https://github.com/marketcalls/historify/blob/master/historify/app/models/dynamic_tables.py"
    click SchedulerJob "https://github.com/marketcalls/historify/blob/master/historify/app/models/scheduler_job.py"
    click SettingsModel "https://github.com/marketcalls/historify/blob/master/historify/app/models/settings.py"
    click StockData "https://github.com/marketcalls/historify/blob/master/historify/app/models/stock_data.py"
    click WatchlistModel "https://github.com/marketcalls/historify/blob/master/historify/app/models/watchlist.py"
    click MainRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/main.py"
    click APIRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/api.py"
    click ChartRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/charts.py"
    click SchedulerRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/scheduler.py"
    click SettingsRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/settings.py"
    click WatchlistRoutes "https://github.com/marketcalls/historify/blob/master/historify/app/routes/watchlist.py"
    click Auth "https://github.com/marketcalls/historify/blob/master/historify/app/utils/auth.py"
    click DataFetcher "https://github.com/marketcalls/historify/blob/master/historify/app/utils/data_fetcher.py"
    click DataFetcherChunked "https://github.com/marketcalls/historify/blob/master/historify/app/utils/data_fetcher_chunked.py"
    click RateLimiter "https://github.com/marketcalls/historify/blob/master/historify/app/utils/rate_limiter.py"
    click SchedulerUtil "https://github.com/marketcalls/historify/blob/master/historify/app/utils/scheduler.py"
    click CSS "https://github.com/marketcalls/historify/blob/master/historify/app/static/css/design-system.css"
    click CSS "https://github.com/marketcalls/historify/blob/master/historify/app/static/css/modal.css"
    click CSS "https://github.com/marketcalls/historify/blob/master/historify/app/static/css/style.css"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/command-palette.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/dashboard.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/download.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/export.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/import.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/scheduler.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/settings.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/sidebar.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/theme.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/tradingview-charts.js"
    click JS "https://github.com/marketcalls/historify/blob/master/historify/app/static/js/watchlist.js"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/base.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/dashboard.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/charts.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/download.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/export.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/import.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/index.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/scheduler.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/settings.html"
    click Templates "https://github.com/marketcalls/historify/blob/master/historify/app/templates/watchlist.html"
    click DocsPRD "https://github.com/marketcalls/historify/blob/master/docs/prd.md"
    click Readme "https://github.com/marketcalls/historify/blob/master/README.md"

    %% Styles
    classDef actor fill:#AEDFF7,stroke:#3498DB,color:#000
    classDef frontend fill:#D6EAF8,stroke:#3498DB,color:#000
    classDef backend fill:#D5F5E3,stroke:#27AE60,color:#000
    classDef model fill:#ABEBC6,stroke:#229954,color:#000
    classDef util fill:#FCF3CF,stroke:#F4D03F,color:#000
    classDef db fill:#FADBD8,stroke:#E74C3C,color:#000
    classDef scheduler fill:#EBDEF0,stroke:#9B59B6,color:#000
    classDef external fill:#E8DAEF,stroke:#8E44AD,color:#000
```
