How umaiszahid/metatrader-python-tick-acquisition works
Project type and purpose
• This is an open-source data-acquisition pipeline/tool that bridges MetaTrader (MT4/MT5) and Python to capture tick data with millisecond precision.
• It uses ZeroMQ in MQL4/MQL5 scripts to publish ticks and Python clients to subscribe, parse, store, and analyze them.
File structure
• Top-level directories:
• “MQL4 Server Connector” & “MQL5 Server Connector” contain the trading-terminal scripts that publish ticks via ZeroMQ.
• “Prototype Python Connector” holds early Python subscriber code (ZeroMQ wrapper, tick I/O, download utilities).
• “Python Client Connector” contains the production-oriented Python subscriber (“python_client_connector.py”) and test harness.
• “Jupyter Notebooks” demonstrates historical data extraction, comparison, and testing workflows.
• Configuration/build: no container or CI files—deploy by loading MQL script into MetaTrader and running the Python client.
Extract additional insights from README
• Primary goal: Capture live ticks from MT5→Python over ZeroMQ with millisecond accuracy.
• Secondary: Convert to JSON, generate OHLC candles, compare against broker history.
• Future goal: Add a pipeline to persist live ticks into a database.
Main components & interactions (
a. MetaTrader Terminal (MT4/MT5)
• Runs the ZeroMQ server connector script (MQL4 or MQL5).
• Publishes raw tick messages (symbol, bid/ask, timestamp) in JSON.
b. ZeroMQ Message Broker
• The PUB socket opened by the MQL script.
• Acts as the transport layer; decouples publisher from subscriber.
c. Python Client Connector
• Subscribes (SUB socket) to ZeroMQ.
• Parses JSON ticks, optionally buffers / transforms (dwx_tick_data_io.py).
• Writes to ephemeral HDF5 or in-memory Pandas DataFrame.
d. Data Pipeline and Storage (future)
• Could be a streaming queue (e.g., Kafka, Redis) or direct DB (TimescaleDB, InfluxDB).
• Persists raw ticks or derived OHLC.
e. Analysis Layer (Jupyter Notebooks)
• Connects to stored data or uses Python connector to backfill historical data.
• Performs comparative analysis against broker-provided historical data.
Architectural patterns & principles
• Pub-Sub messaging (ZeroMQ) for loose coupling.
• Bridge pattern: abstracts MetaTrader specifics (MQL) from Python analytics.
• ETL pipeline: Extract (MQL), Transform (JSON, Pandas), Load (HDF5/DB).
• Modular separation: trading-client (MQL) vs. processing-client (Python).
Final advice
• Be exhaustive: each script folder is its own deployable/publishable component.
• Both current and planned (“To Do”) parts for roadmap clarity.