How marketcalls/historify works
Project Type and Purpose
• This is a full-stack web application (“Historify”) for managing, visualizing, and scheduling historical stock market data downloads.
• Backend: Flask + SQLAlchemy + SQLite + APScheduler
• Frontend: Jinja2 templates + Tailwind CSS/DaisyUI + Vanilla ES6 JavaScript + TradingView Charts
File Structure Analysis
• Top-level:
• run.py – application entry point
• requirements.txt – dependencies
• README.md, docs/prd.md – high-level docs
• historify/app/:
• models/ – SQLAlchemy models & dynamic table definitions
• routes/ – Flask view functions (API endpoints + page routes)
• utils/ – data_fetcher, rate_limiter, scheduler helpers
• static/ – CSS, JS, images (design system, TradingView integration)
• templates/ – Jinja2 pages (dashboard, import/export, charts, scheduler)
• historify/.env.sample – environment configuration (API keys, DB URI, secrets)
Key Architectural Components
a. Frontend Layer
• Browser UI (HTML templates + CSS design system + JS modules)
• TradingView Lightweight Charts loaded client-side
• Command Palette, modals, tables, forms for import/export/watchlist/scheduler
b. Backend Layer
• Flask Application
– Routes split into API (JSON) vs. Page renderers
– Blueprints in app/routes/api.py, /charts.py, /scheduler.py, etc.
• Utils
– DataFetcher (bulk download, parallel threads, rate-limit)
– Scheduler (APScheduler jobs, IST timezone handling)
• Models
– Dynamic tables per symbol/interval
– Checkpoint, Watchlist, SchedulerJob, Settings
• Task Queue / Background Processing
– APScheduler for scheduled jobs
– ThreadPoolExecutor for parallel downloads
c. Database
• SQLite (DATABASE_URI from .env)
• SQLAlchemy ORM + dynamic table creation for efficient time-series storage
d. External Services
• OpenAlgo API (market data)
• TradingView JS library (client-side chart rendering)
Relationships and Data Flows
• User interacts with Browser UI → JS calls REST API endpoints (e.g., /api/download, /api/import-symbols)
• Flask routes handle requests → use utils/DataFetcher to call OpenAlgo API → write/read to SQLite via SQLAlchemy models
• Scheduler subsystem schedules jobs via APScheduler → triggers DataFetcher tasks in background → updates database
• Export requests queue CSV generation → streams result to user, monitors status via /api/export/queue
• Real-time watchlist/quotes: JS polling / GET /api/quotes → Flask returns JSON from models or external API
Architectural Patterns and Principles
• Separation of concerns: UI templates vs. API endpoints vs. data/model layer vs. scheduler services
• Modular routing using Flask Blueprints
• Dynamic table creation for scale (one table per symbol/interval)
• Background job scheduling (APScheduler) and parallel processing (ThreadPool)
• Rate limiting at utility level to respect API quotas
• Lazy loading / virtual scrolling for performance in UI
Essential Elements to Capture
• Frontend Modules: CSS design system, JS (dashboard.js, import.js, watchlist.js, scheduler.js, charts.js)
• Flask Blueprints: API vs. Page routes
• Utils: DataFetcher, RateLimiter, Scheduler helpers
• Scheduler Manager: APScheduler jobs configured in utils/scheduler.py
• Dynamic Data Model: Symbol/interval tables, checkpoint model
• Export Queue: background CSV streaming
• External Integrations: OpenAlgo (data), TradingView (charts)
Final Recommendations
• Don’t over-split: group similar utilities together (e.g., all data-fetching logic)
• Asynchronous/background flows separately from synchronous HTTP flows