How erikhafnor/mtudatalake works
Project Type & Purpose
• Type: Backend-only ingestion service (no GUI).
• Purpose: Receive HL7 v2.x messages over MLLP, parse & flatten OBX segments, tag with device serial, and stream into QuestDB (or CSV) for predictive maintenance analytics.
a. External HL7 Sender
• Description: Any medical device or simulator (e.g., HAPI TestPanel) that opens an MLLP TCP connection.
b. HL7 MLLP Receiver (hl7_mllp_receiver.py)
• Listens on TCP port 2554, extracts raw HL7 frames, sends ACKs.
• Dependency: Python standard library’s socket, threading.
c. Parser & Enricher (parsing.py + variable_mapping.py)
• Flattens OBX segments and bitfields per variable_mapping.py.
• Tracks last serial for backfill logic.
• Sends to two sinks: QuestDB via ILP & optional CSV.
d. Ingestion Sinks
1. QuestDB (ILP on port 9009)
– High-performance time-series DB for downstream ML.
2. CSV Export (optional)
– Flat files (for debugging or batch analysis via pandas).
e. Admin/User Interfaces
• QuestDB Web Console (port 9000) for SQL queries & dashboards.
• Optional scripts or pandas notebooks.
Architectural Patterns & Design Principles
• Pipeline pattern: sequential stages (Receive → Parse → Store).
• Single responsibility: each script handles one concern (network, parsing, mapping, ingestion).
• Configurability: variable_mapping.py and inline constants in receiver for ports, app/fac names.
• Fault tolerance: automatic ACKs, serial backfill for missing OBX.
Technologies & Frameworks
• Python 3.x standard library (socket, threading, datetime).
• QuestDB (ILP ingestion, Web Console).
• Optional: pandas for CSV export, python-dotenv for environment overrides.
Detailed Component Breakdown (optional sub-components)
• HL7 MLLP Receiver
• Socket listener
• Frame parser & ACK generator
• Parser Module
• HL7 message loader
• OBX extractor
• Bitfield flattener
• Serial backfill logic
• ILP line builder
• Variable Mapping
• JSON/dict definitions for variable IDs & bitfields
• CSV Exporter
• pandas or CSV writer hook