# umaiszahid/metatrader-python-tick-acquisition architecture

> Architecture diagram of the GitHub repository umaiszahid/metatrader-python-tick-acquisition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/umaiszahid/metatrader-python-tick-acquisition
- Repository: https://github.com/umaiszahid/metatrader-python-tick-acquisition

## Overview

##### Project type and purpose
- This is an open-source data-acquisition pipeline/tool that bridges MetaTrader (MT4/MT5) and Python to capture tick data with millisecond precision.
- It uses ZeroMQ in MQL4/MQL5 scripts to publish ticks and Python clients to subscribe, parse, store, and analyze them.

##### File structure
- Top-level directories:
- “MQL4 Server Connector” & “MQL5 Server Connector” contain the trading-terminal scripts that publish ticks via ZeroMQ.
- “Prototype Python Connector” holds early Python subscriber code (ZeroMQ wrapper, tick I/O, download utilities).
- “Python Client Connector” contains the production-oriented Python subscriber (“python_client_connector.py”) and test harness.
- “Jupyter Notebooks” demonstrates historical data extraction, comparison, and testing workflows.
- Configuration/build: no container or CI files—deploy by loading MQL script into MetaTrader and running the Python client.

##### Extract additional insights from README
- Primary goal: Capture live ticks from MT5→Python over ZeroMQ with millisecond accuracy.
- Secondary: Convert to JSON, generate OHLC candles, compare against broker history.
- Future goal: Add a pipeline to persist live ticks into a database.

##### Main components & interactions (
a. MetaTrader Terminal (MT4/MT5)
- Runs the ZeroMQ server connector script (MQL4 or MQL5).
- Publishes raw tick messages (symbol, bid/ask, timestamp) in JSON.
b. ZeroMQ Message Broker
- The PUB socket opened by the MQL script.
- Acts as the transport layer; decouples publisher from subscriber.
c. Python Client Connector
- Subscribes (SUB socket) to ZeroMQ.
- Parses JSON ticks, optionally buffers / transforms (dwx_tick_data_io.py).
- Writes to ephemeral HDF5 or in-memory Pandas DataFrame.
d. Data Pipeline and Storage (future)
- Could be a streaming queue (e.g., Kafka, Redis) or direct DB (TimescaleDB, InfluxDB).
- Persists raw ticks or derived OHLC.
e. Analysis Layer (Jupyter Notebooks)
- Connects to stored data or uses Python connector to backfill historical data.
- Performs comparative analysis against broker-provided historical data.

##### Architectural patterns & principles
- Pub-Sub messaging (ZeroMQ) for loose coupling.
- Bridge pattern: abstracts MetaTrader specifics (MQL) from Python analytics.
- ETL pipeline: Extract (MQL), Transform (JSON, Pandas), Load (HDF5/DB).
- Modular separation: trading-client (MQL) vs. processing-client (Python).

##### Final advice
- Be exhaustive: each script folder is its own deployable/publishable component.
- Both current and planned (“To Do”) parts for roadmap clarity.

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Swimlanes
  subgraph "Trading Terminal Layer"
    direction TB
    MQL4_Pub["MQL4 ZeroMQ Publisher"]:::component
    MQL4_Orig["MQL4 Original Server Connector"]:::component
    MQL5_Pub["MQL5 ZeroMQ Publisher"]:::component
  end

  subgraph "Messaging Layer"
    direction TB
    ZeroMQ_Broker["ZeroMQ Broker (PUB/SUB)"]:::broker
  end

  subgraph "Data Processing & Analysis Layer"
    direction TB

    subgraph "Prototype Python Connector"
      direction TB
      ZeroMQ_wrapper["ZeroMQ Wrapper"]:::component
      Tick_IO["Tick I/O Module"]:::component
      Download_Utility["Historical Download Utility"]:::component
    end

    subgraph "Production Python Connector"
      direction TB
      Python_Client["Python Client Connector"]:::component
      Test_Harness["Test Harness"]:::component
    end

    Storage["Future Storage/DB"]:::storage

    subgraph "Analysis Layer (Notebooks)"
      direction TB
      Notebook_HistComp["Historical Data Extraction & Comparison.ipynb"]:::notebook
      Notebook_TestMQL5["Testing Subscription Tick Data - MQL5.ipynb"]:::notebook
      Notebook_TestSub["Testing Subscription Tick Data.ipynb"]:::notebook
      Notebook_TestOther["Testing.ipynb"]:::notebook
      TestHDF5["Sample HDF5 Data File"]:::storage
    end
  end

  %% Data Flows
  MQL4_Pub -->|"Publishes JSON ticks @ ms precision"| ZeroMQ_Broker
  MQL4_Orig -->|"Publishes JSON ticks @ ms precision"| ZeroMQ_Broker
  MQL5_Pub -->|"Publishes JSON ticks @ ms precision"| ZeroMQ_Broker

  ZeroMQ_Broker -->|"SUB socket"| ZeroMQ_wrapper
  ZeroMQ_Broker -->|"SUB socket"| Python_Client

  ZeroMQ_wrapper -->|"Parses & streams ticks"| Tick_IO
  ZeroMQ_wrapper -->|"Downloads historical data"| Download_Utility

  Python_Client -->|"Transforms (Pandas OHLC)"| Storage
  Tick_IO -->|"In-memory DataFrame / HDF5"| Storage

  Python_Client -->|"Provides live data"| Notebook_HistComp
  Storage -->|"Backfill & analysis"| Notebook_HistComp
  Python_Client -->|"Test ticks"| Notebook_TestMQL5
  Python_Client -->|"Test ticks"| Notebook_TestSub
  Python_Client -->|"General tests"| Notebook_TestOther
  Storage -->|"Sample data"| TestHDF5

  %% Documentation
  README["README.md"]:::component
  TODO["TODO.txt"]:::component
  README --> Notebook_HistComp
  TODO --> Notebook_TestOther

  %% Click Events
  click MQL4_Pub "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/MQL4 Server Connector/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4"
  click MQL4_Orig "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/MQL4 Server Connector/Server Connector Original.mq4"
  click MQL5_Pub "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/MQL5 Server Connector/MQL Server ZeroMQ Connector.mq5"
  click ZeroMQ_wrapper "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Prototype Python Connector/ZeroMQ Python Connector.py"
  click Tick_IO "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Prototype Python Connector/dwx_tick_data_io.py"
  click Download_Utility "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Prototype Python Connector/dwx_tickdata_download.py"
  click Python_Client "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Python Client Connector/python_client_connector.py"
  click Test_Harness "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Python Client Connector/Client Testing.py"
  click Notebook_HistComp "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Jupyter Notebooks/Historical Data Extraction and Comparison.ipynb"
  click Notebook_TestMQL5 "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Jupyter Notebooks/Testing Subscription Tick Data - MQL5.ipynb"
  click Notebook_TestSub "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Jupyter Notebooks/Testing Subscription Tick Data.ipynb"
  click Notebook_TestOther "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Jupyter Notebooks/Testing.ipynb"
  click TestHDF5 "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/Jupyter Notebooks/testTickData.h5"
  click README "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/README.md"
  click TODO "https://github.com/umaiszahid/metatrader-python-tick-acquisition/blob/master/TODO.txt"

  %% Styles
  classDef component fill:#D6E8FF,stroke:#333,stroke-width:1px
  classDef broker fill:#C2F0C2,stroke:#333,stroke-width:1px
  classDef storage fill:#FFD8A8,stroke:#333,stroke-width:1px
  classDef notebook fill:#FFF2CC,stroke:#333,stroke-width:1px
```
