# erikhafnor/mtudatalake architecture

> Architecture diagram of the GitHub repository erikhafnor/mtudatalake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/erikhafnor/mtudatalake
- Repository: https://github.com/erikhafnor/mtudatalake

## Overview

##### Project Type & Purpose
- Type: Backend-only ingestion service (no GUI).
- Purpose: Receive HL7 v2.x messages over MLLP, parse & flatten OBX segments, tag with device serial, and stream into QuestDB (or CSV) for predictive maintenance analytics.

a. External HL7 Sender
- Description: Any medical device or simulator (e.g., HAPI TestPanel) that opens an MLLP TCP connection.
b. HL7 MLLP Receiver (hl7_mllp_receiver.py)
- Listens on TCP port 2554, extracts raw HL7 frames, sends ACKs.
- Dependency: Python standard library’s socket, threading.
c. Parser & Enricher (parsing.py + variable_mapping.py)
- Flattens OBX segments and bitfields per variable_mapping.py.
- Tracks last serial for backfill logic.
- Sends to two sinks: QuestDB via ILP & optional CSV.
d. Ingestion Sinks
1. QuestDB (ILP on port 9009)
– High-performance time-series DB for downstream ML.
2. CSV Export (optional)
– Flat files (for debugging or batch analysis via pandas).
e. Admin/User Interfaces
- QuestDB Web Console (port 9000) for SQL queries & dashboards.
- Optional scripts or pandas notebooks.

##### Architectural Patterns & Design Principles
- Pipeline pattern: sequential stages (Receive → Parse → Store).
- Single responsibility: each script handles one concern (network, parsing, mapping, ingestion).
- Configurability: variable_mapping.py and inline constants in receiver for ports, app/fac names.
- Fault tolerance: automatic ACKs, serial backfill for missing OBX.

##### Technologies & Frameworks
- Python 3.x standard library (socket, threading, datetime).
- QuestDB (ILP ingestion, Web Console).
- Optional: pandas for CSV export, python-dotenv for environment overrides.

##### Detailed Component Breakdown (optional sub-components)
- HL7 MLLP Receiver
- Socket listener
- Frame parser & ACK generator
- Parser Module
- HL7 message loader
- OBX extractor
- Bitfield flattener
- Serial backfill logic
- ILP line builder
- Variable Mapping
- JSON/dict definitions for variable IDs & bitfields
- CSV Exporter
- pandas or CSV writer hook

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        HL7Sender["External HL7 Sender"]:::external
    end

    %% Configuration
    Env["Configuration (.env)"]:::config
    click Env "https://github.com/erikhafnor/mtudatalake/blob/main/python_dotenv-1.1.1-py3-none-any.whl"

    %% Network/Listening Layer
    RunServer["Server Startup\n(run_server.py)"]:::network
    click RunServer "https://github.com/erikhafnor/mtudatalake/blob/main/run_server.py"

    Receiver["HL7 MLLP Receiver\n(hl7_mllp_receiver.py)"]:::network
    click Receiver "https://github.com/erikhafnor/mtudatalake/blob/main/hl7_mllp_receiver.py"

    %% Processing Layer
    subgraph "Processing Layer"
        Parsing["Parser & Enricher\n(parsing.py)"]:::processing
        VarMap["Variable Mapping\n(variable_mapping.py)"]:::processing
    end
    click Parsing "https://github.com/erikhafnor/mtudatalake/blob/main/parsing.py"
    click VarMap "https://github.com/erikhafnor/mtudatalake/blob/main/variable_mapping.py"

    %% Storage Layer
    QuestDB["QuestDB (ILP 9009)"]:::storage
    CSV["CSV Export"]:::storage

    %% User Interface Layer
    WebConsole["QuestDB Web Console\n(port 9000)"]:::ui

    %% Documentation
    subgraph "Documentation"
        FieldManual["Field Manual"]:::docs
        README["README.md"]:::docs
        LICENSE["LICENSE"]:::docs
    end
    click FieldManual "https://github.com/erikhafnor/mtudatalake/blob/main/Field_Manual_MTUdatalake.html"
    click README "https://github.com/erikhafnor/mtudatalake/blob/main/README.md"
    click LICENSE "https://github.com/erikhafnor/mtudatalake/tree/main/LICENSE"

    %% Data Flows
    HL7Sender -->|"TCP MLLP 2554"| Receiver
    RunServer --> Receiver
    Env --> Receiver
    Receiver -->|"HL7 Msg & Control ID"| Parsing
    Env --> Parsing
    VarMap -.-> Parsing
    Parsing -->|"ILP Protocol"| QuestDB
    Parsing -->|"CSV Export"| CSV
    QuestDB -->|"SQL Queries"| WebConsole

    %% Styles
    classDef external fill:#AEDFFD,stroke:#0366d6,color:#0366d6;
    classDef network fill:#D4F8CC,stroke:#2E8B57,color:#2E8B57;
    classDef processing fill:#FFF4B1,stroke:#D4A017,color:#D4A017;
    classDef storage fill:#FFD8A9,stroke:#FF8C00,color:#FF8C00;
    classDef ui fill:#E2D3F5,stroke:#800080,color:#800080;
    classDef config fill:#D0F0C0,stroke:#20B2AA,color:#20B2AA;
    classDef docs fill:#E0E0E0,stroke:#A9A9A9,color:#696969;
```
