How walkerad03/mpsc works
Project type and purpose
• This is a data‐processing and machine‐learning pipeline project. Its primary goal is to extract raw clinical or observational data via scripts, ingest and parse that data into a standardized format, then run exploratory analyses and model training in Jupyter notebooks.
File structure
• Top‐level files:
• main.py: likely the entry point/orchestrator.
• pyproject.toml & .python-version: project configuration and environment.
• README.md: currently empty—no built‐in architecture doc.
• Directories:
• scripts/: Extraction scripts (GetBPObs.py, GetWeightObs.py, etc.) that pull different observation types from source systems or JSON files.
• src/ingestion/: Core ingestion framework—dispatcher.py routes raw inputs to parser modules; ingest_json.py reads JSON; parsers/ contains per-domain logic (demographics, enteral_feed, generic_parser).
• src/pipeline.py: Orchestrates end-to-end ETL (calls dispatcher, applies transformations, writes outputs).
• notebooks/: Data‐science layer for EDA (eda.ipynb), hyperparameter tuning (hyperparams.ipynb), random forest modeling, and final results.
• resources/: Static data files or lookup tables (FluidIntakeRowIDs.txt, etc.).
README
• No content provided. We rely entirely on the structure and file naming conventions.
a. Main components:
• Raw Data & Configuration:
• scripts/ directory (custom extraction scripts)
• resources/ static lookup files
• Ingestion & Parsing:
• ingest_json.py (JSON reader)
• dispatcher.py (routes to parsers)
• parsers/ (demographics, enteral_feed, generic_parser)
• Pipeline Orchestrator:
• pipeline.py invoked by main.py
• Data Storage:
• Intermediate outputs (e.g., UnaccumulatedFluids.txt)
• Final processed datasets
• Data Science Layer:
• notebooks/ for EDA and modeling
b. Relationships & interactions:
1. Extraction scripts read external sources → produce raw JSON or flat files.
2. ingest_json.py loads the JSON into memory.
3. dispatcher.py invokes the appropriate parser module.
4. Parser modules transform raw records into standardized data rows.
5. pipeline.py coordinates the sequence: extraction → ingestion → parsing → output generation.
6. Outputs are stored on disk under resources/ or a designated output folder.
7. Jupyter notebooks read these processed datasets for analysis and ML modeling.
c. Architectural patterns & principles:
• Modular ETL: Separation of extraction (scripts/) vs. transformation (parsers/) vs. orchestration (pipeline.py).
• Plugin‐style parsers: New data domains can be supported by adding parser modules to parsers/.
• Clear layering: Data ingestion layer vs. data science layer.
d. Key technologies & frameworks:
• Python standard libs and custom scripts
• Jupyter notebooks for analysis
• pyproject.toml for dependency management (likely Poetry or similar)