# walkerad03/mpsc architecture

> Architecture diagram of the GitHub repository walkerad03/mpsc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/walkerad03/mpsc
- Repository: https://github.com/walkerad03/mpsc

## Overview

##### Project type and purpose
- This is a data‐processing and machine‐learning pipeline project. Its primary goal is to extract raw clinical or observational data via scripts, ingest and parse that data into a standardized format, then run exploratory analyses and model training in Jupyter notebooks.

##### File structure
- Top‐level files:
- main.py: likely the entry point/orchestrator.
- pyproject.toml & .python-version: project configuration and environment.
- README.md: currently empty—no built‐in architecture doc.
- Directories:
- scripts/: Extraction scripts (GetBPObs.py, GetWeightObs.py, etc.) that pull different observation types from source systems or JSON files.
- src/ingestion/: Core ingestion framework—dispatcher.py routes raw inputs to parser modules; ingest_json.py reads JSON; parsers/ contains per-domain logic (demographics, enteral_feed, generic_parser).
- src/pipeline.py: Orchestrates end-to-end ETL (calls dispatcher, applies transformations, writes outputs).
- notebooks/: Data‐science layer for EDA (eda.ipynb), hyperparameter tuning (hyperparams.ipynb), random forest modeling, and final results.
- resources/: Static data files or lookup tables (FluidIntakeRowIDs.txt, etc.).

##### README
- No content provided. We rely entirely on the structure and file naming conventions.

a. Main components:
- Raw Data & Configuration:
- scripts/ directory (custom extraction scripts)
- resources/ static lookup files
- Ingestion & Parsing:
- ingest_json.py (JSON reader)
- dispatcher.py (routes to parsers)
- parsers/ (demographics, enteral_feed, generic_parser)
- Pipeline Orchestrator:
- pipeline.py invoked by main.py
- Data Storage:
- Intermediate outputs (e.g., UnaccumulatedFluids.txt)
- Final processed datasets
- Data Science Layer:
- notebooks/ for EDA and modeling
b. Relationships & interactions:
1. Extraction scripts read external sources → produce raw JSON or flat files.
2. ingest_json.py loads the JSON into memory.
3. dispatcher.py invokes the appropriate parser module.
4. Parser modules transform raw records into standardized data rows.
5. pipeline.py coordinates the sequence: extraction → ingestion → parsing → output generation.
6. Outputs are stored on disk under resources/ or a designated output folder.
7. Jupyter notebooks read these processed datasets for analysis and ML modeling.
c. Architectural patterns & principles:
- Modular ETL: Separation of extraction (scripts/) vs. transformation (parsers/) vs. orchestration (pipeline.py).
- Plugin‐style parsers: New data domains can be supported by adding parser modules to parsers/.
- Clear layering: Data ingestion layer vs. data science layer.
d. Key technologies & frameworks:
- Python standard libs and custom scripts
- Jupyter notebooks for analysis
- pyproject.toml for dependency management (likely Poetry or similar)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Extraction Layer
    subgraph "Extraction Scripts"
        direction TB
        E1["GetAllParameters.py"]:::extraction
        E2["GetBPObs.py"]:::extraction
        E3["GetCalculatedEnergyObs.py"]:::extraction
        E4["GetDemographics.py"]:::extraction
        E5["GetEnteralFeedObs.py"]:::extraction
        E6["GetFluids.py"]:::extraction
        E7["GetHeadCircumObs.py"]:::extraction
        E8["GetHeightObs.py"]:::extraction
        E9["GetPulseObs.py"]:::extraction
        E10["GetRespObs.py"]:::extraction
        E11["GetSP02.py"]:::extraction
        E12["GetStoolWeight.py"]:::extraction
        E13["GetWeightObs.py"]:::extraction
        E14["convert_json_data.py"]:::extraction
        E15["hero_fsdb.py"]:::extraction
    end

    %% Configuration Files
    subgraph "Configuration"
        direction TB
        C1[".python-version"]:::config
        C2["pyproject.toml"]:::config
    end

    %% Ingestion Engine
    subgraph "Ingestion Engine"
        direction TB
        I1["ingest_json.py"]:::ingestion
        I2["dispatcher.py"]:::ingestion
    end

    %% Parser Modules
    subgraph "Parsers"
        direction TB
        P1["demographics.py"]:::parser
        P2["enteral_feed.py"]:::parser
        P3["generic_parser.py"]:::parser
    end

    %% Pipeline Orchestrator
    subgraph "Pipeline Orchestrator"
        direction TB
        O1["main.py"]:::orchestrator
        O2["pipeline.py"]:::orchestrator
    end

    %% Data Storage / Resources
    subgraph "Data Storage / Static Resources"
        direction TB
        S1["FluidIntakeRowIDs.txt"]:::storage
        S2["FluidOutputRowIDs.txt"]:::storage
        S3["UnaccumulatedFluids.txt"]:::storage
    end

    %% Data Science Layer
    subgraph "Data Science Layer"
        direction TB
        N1["data_consolidation.ipynb"]:::notebook
        N2["eda.ipynb"]:::notebook
        N3["hyperparams.ipynb"]:::notebook
        N4["random_forest.ipynb"]:::notebook
        N5["final.ipynb"]:::notebook
    end

    %% Connections
    C1 -->|configures| O1
    C2 -->|configures| O1
    O1 -->|"calls"| O2
    O2 -->|"runs extraction scripts"| E1
    O2 -->|"runs extraction scripts"| E2
    O2 -->|"runs extraction scripts"| E3
    O2 -->|"runs extraction scripts"| E4
    O2 -->|"runs extraction scripts"| E5
    O2 -->|"runs extraction scripts"| E6
    O2 -->|"runs extraction scripts"| E7
    O2 -->|"runs extraction scripts"| E8
    O2 -->|"runs extraction scripts"| E9
    O2 -->|"runs extraction scripts"| E10
    O2 -->|"runs extraction scripts"| E11
    O2 -->|"runs extraction scripts"| E12
    O2 -->|"runs extraction scripts"| E13
    O2 -->|"runs extraction scripts"| E14
    O2 -->|"runs extraction scripts"| E15

    E1 -->|"JSON/TXT"| I1
    E2 -->|"JSON/TXT"| I1
    E3 -->|"JSON/TXT"| I1
    E4 -->|"JSON/TXT"| I1
    E5 -->|"JSON/TXT"| I1
    E6 -->|"JSON/TXT"| I1
    E7 -->|"JSON/TXT"| I1
    E8 -->|"JSON/TXT"| I1
    E9 -->|"JSON/TXT"| I1
    E10 -->|"JSON/TXT"| I1
    E11 -->|"JSON/TXT"| I1
    E12 -->|"JSON/TXT"| I1
    E13 -->|"JSON/TXT"| I1
    E14 -->|"JSON/TXT"| I1
    E15 -->|"external DB"| I1

    I1 -->|"records"| I2
    I2 -->|"dispatches to parsers"| P1
    I2 -->|"dispatches to parsers"| P2
    I2 -->|"dispatches to parsers"| P3

    P1 -->|"standardized rows"| S1
    P2 -->|"standardized rows"| S2
    P3 -->|"standardized rows"| S3

    S1 -->|"inputs for notebooks"| N1
    S2 -->|"inputs for notebooks"| N2
    S3 -->|"inputs for notebooks"| N3
    S1 -->|"inputs for notebooks"| N4
    S2 -->|"inputs for notebooks"| N5

    %% Click Events
    click E1 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetAllParameters.py"
    click E2 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetBPObs.py"
    click E3 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetCalculatedEnergyObs.py"
    click E4 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetDemographics.py"
    click E5 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetEnteralFeedObs.py"
    click E6 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetFluids.py"
    click E7 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetHeadCircumObs.py"
    click E8 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetHeightObs.py"
    click E9 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetPulseObs.py"
    click E10 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetRespObs.py"
    click E11 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetSP02.py"
    click E12 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetStoolWeight.py"
    click E13 "https://github.com/walkerad03/mpsc/blob/main/scripts/GetWeightObs.py"
    click E14 "https://github.com/walkerad03/mpsc/blob/main/scripts/convert_json_data.py"
    click E15 "https://github.com/walkerad03/mpsc/blob/main/scripts/hero_fsdb.py"
    click C1 "https://github.com/walkerad03/mpsc/blob/main/.python-version"
    click C2 "https://github.com/walkerad03/mpsc/blob/main/pyproject.toml"
    click I1 "https://github.com/walkerad03/mpsc/blob/main/src/ingestion/ingest_json.py"
    click I2 "https://github.com/walkerad03/mpsc/blob/main/src/ingestion/dispatcher.py"
    click P1 "https://github.com/walkerad03/mpsc/blob/main/src/ingestion/parsers/demographics.py"
    click P2 "https://github.com/walkerad03/mpsc/blob/main/src/ingestion/parsers/enteral_feed.py"
    click P3 "https://github.com/walkerad03/mpsc/blob/main/src/ingestion/parsers/generic_parser.py"
    click O1 "https://github.com/walkerad03/mpsc/blob/main/main.py"
    click O2 "https://github.com/walkerad03/mpsc/blob/main/src/pipeline.py"
    click S1 "https://github.com/walkerad03/mpsc/blob/main/resources/FluidIntakeRowIDs.txt"
    click S2 "https://github.com/walkerad03/mpsc/blob/main/resources/FluidOutputRowIDs.txt"
    click S3 "https://github.com/walkerad03/mpsc/blob/main/resources/UnaccumulatedFluids.txt"
    click N1 "https://github.com/walkerad03/mpsc/blob/main/notebooks/data_consolidation.ipynb"
    click N2 "https://github.com/walkerad03/mpsc/blob/main/notebooks/eda.ipynb"
    click N3 "https://github.com/walkerad03/mpsc/blob/main/notebooks/hyperparams.ipynb"
    click N4 "https://github.com/walkerad03/mpsc/blob/main/notebooks/random_forest.ipynb"
    click N5 "https://github.com/walkerad03/mpsc/blob/main/notebooks/final.ipynb"

    %% Styles
    classDef extraction fill:#AED6F1,stroke:#21618C,color:#1B4F72
    classDef config fill:#D6EAF8,stroke:#2874A6,color:#1B4F72
    classDef ingestion fill:#FCF3CF,stroke:#B7950B,color:#7D6608
    classDef parser fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef orchestrator fill:#FADBD8,stroke:#C0392B,color:#641E16
    classDef storage fill:#E5E8E8,stroke:#626567,color:#1C2833
    classDef notebook fill:#F9EBEA,stroke:#922B21,color:#641E16
```
