# finmind/finmind-visualization architecture

> Architecture diagram of the GitHub repository finmind/finmind-visualization, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/finmind/finmind-visualization
- Repository: https://github.com/finmind/finmind-visualization

## Overview

##### Project type and purpose
- This is a data‐pipeline / microservices‐style backend system that ingests financial data from the external FinMind API, processes and stores it, then exposes it via REST endpoints and a lightweight visualization service. It also provides an analytics/dashboard layer via Redash.

##### File structure
- Top level: Docker orchestration (Dockerfile, make targets), environment configs (mysql.yml, rabbitmq.yml, redash.yml, finmind.yml, local.ini), Python packaging (setup.py, Pipfile).
- “visualization/”: core Python code
- api/v1: FastAPI endpoints for creating tables, syncing history, updating data
- backend/: application setup, routers, DB client, FinMind client
- scheduler.py: job scheduler that enqueues periodic tasks
- tasks/: Celery task definitions and worker bootstrap
- schema/: input schemas
- utility/: shared helper code
- tests/: unit tests for the task layer

##### README insights
- Services: MySQL, RabbitMQ (broker), Flower (monitoring), visualization (UI at :5000), finmind-visualization-api (FastAPI at :8888), Redash
- Make targets to spin up infrastructure and the FinMind sync API (with or without token)

##### Detail
- Side‐services (Flower monitoring)
- Retry / error‐handling loops if relevant (Celery retries)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Pipeline
    subgraph "Ingestion Pipeline"
        classDef ingestion fill:#D0E8FF,stroke:#003366;
        Scheduler["Scheduler Process"]:::ingestion
        RabbitMQ["RabbitMQ Broker"]:::ingestion
        Worker["Celery Worker"]:::ingestion
        TaskDefs["Celery Task Definitions"]:::ingestion
    end

    %% External API
    subgraph "External Services"
        classDef external fill:#FFE0E0,stroke:#660000;
        FinMindAPI["FinMind API"]:::external
    end

    %% Storage
    subgraph "Storage"
        classDef storage fill:#E0FFD8,stroke:#006600;
        MySQLDB["MySQL Database"]:::storage
    end

    %% API Layer
    subgraph "API Layer"
        classDef api fill:#FFE8CC,stroke:#994C00;
        FastAPI["FastAPI Server :8888"]:::api
        APIv1["API v1"]:::api
        CreateTable["Create Table Endpoint"]:::api
        SyncHistory["Sync History Endpoint"]:::api
        UpdateData["Update Data Endpoint"]:::api
    end

    %% Presentation
    subgraph "Presentation"
        classDef presentation fill:#F2E0FF,stroke:#660066;
        VizUI["Visualization UI :5000"]:::presentation
        Redash["Redash Dashboard"]:::presentation
    end

    %% Monitoring
    subgraph "Monitoring"
        classDef monitoring fill:#F0F0F0,stroke:#888888;
        Flower["Flower Monitoring UI"]:::monitoring
    end

    %% Configurations (footnote style)
    subgraph "Configurations"
        classDef config fill:#FFFFFF,stroke:#000000,stroke-dasharray: 2 2;
        MQConfig["RabbitMQ Config"]:::config
        MySQLConfig["MySQL Config"]:::config
        RedashConfig["Redash Config"]:::config
        GlobalConfig["Global Config Loader"]:::config
    end

    %% Connections
    Scheduler -->|"publish tasks"| RabbitMQ
    RabbitMQ -->|"consume tasks"| Worker
    Worker -->|"execute tasks"| TaskDefs
    Worker -->|"call API"| FinMindAPI
    Worker -->|"write data"| MySQLDB
    FastAPI -->|"read/write"| MySQLDB
    VizUI -->|"HTTP"| FastAPI
    Redash -->|"SQL queries"| MySQLDB
    Flower -->|"monitor"| RabbitMQ
    Flower -->|"monitor"| Worker

    %% Config connections
    MQConfig --> RabbitMQ
    MySQLConfig --> MySQLDB
    RedashConfig --> Redash
    GlobalConfig --> FastAPI
    GlobalConfig --> Worker

    %% Click Events
    click Scheduler "https://github.com/finmind/finmind-visualization/blob/master/visualization/scheduler.py"
    click MQConfig "https://github.com/finmind/finmind-visualization/blob/master/rabbitmq.yml"
    click Worker "https://github.com/finmind/finmind-visualization/blob/master/visualization/tasks/worker.py"
    click TaskDefs "https://github.com/finmind/finmind-visualization/blob/master/visualization/tasks/task.py"
    click FinMindAPI "https://github.com/finmind/finmind-visualization/blob/master/visualization/backend/finmind.py"
    click MySQLDB "https://github.com/finmind/finmind-visualization/blob/master/visualization/backend/database.py"
    click FastAPI "https://github.com/finmind/finmind-visualization/blob/master/visualization/main.py"
    click APIv1 "https://github.com/finmind/finmind-visualization/blob/master/visualization/api/v1/api.py"
    click CreateTable "https://github.com/finmind/finmind-visualization/blob/master/visualization/api/v1/endpoints/create_mysql_table.py"
    click SyncHistory "https://github.com/finmind/finmind-visualization/blob/master/visualization/api/v1/endpoints/sync_history_data.py"
    click UpdateData "https://github.com/finmind/finmind-visualization/blob/master/visualization/api/v1/endpoints/update_data.py"
    click GlobalConfig "https://github.com/finmind/finmind-visualization/blob/master/visualization/config.py"
    click MySQLConfig "https://github.com/finmind/finmind-visualization/blob/master/mysql.yml"
    click RedashConfig "https://github.com/finmind/finmind-visualization/blob/master/redash.yml"

    %% Styles
    class Scheduler,RabbitMQ,Worker,TaskDefs ingestion
    class FinMindAPI external
    class MySQLDB storage
    class FastAPI,APIv1,CreateTable,SyncHistory,UpdateData api
    class VizUI,Redash presentation
    class Flower monitoring
    class MQConfig,MySQLConfig,RedashConfig,GlobalConfig config
```
