How dermatologist/fhiry works
Project Type and Purpose
• This is a Python library (package) called fhiry that provides a virtual, flattened view of FHIR (Fast Healthcare Interoperability Resources) data in pandas DataFrames for analytics, AI/ML, and LLM‐based queries.
• It supports four primary data sources:
• A folder of FHIR Bundle JSON files
• A folder of NDJSON files (bulk data export)
• A FHIR REST API (FHIR Search)
• Google BigQuery FHIR tables
File Structure
• Top‐level directories
• src/fhiry/ – core modules:
– base_fhiry.py – common base classes and utilities
– parallel.py – orchestrates parallel processing of files
– fhirndjson.py – NDJSON ingestion and flattening
– fhirsearch.py – FHIR search API client and flattening
– bqsearch.py – BigQuery client adapter and flattening
– flattenfhir.py – logic to flatten a FHIR Bundle to a tabular form
• examples/ – usage scripts (e.g., LLM example)
• notebooks/ – Jupyter notebooks demonstrating workflows
• tests/ – unit tests and test resources
• docs/ – Sphinx documentation site
• Configuration & CI
• pyproject.toml/setup.py – packaging and dependencies
• .github/workflows/ / tox.ini / .devcontainer/ – CI builds, tests, and dev environment
README for Additional Insights
• Describes four ingestion workflows (Bundles, NDJSON, FHIR Search, BigQuery)
• Mentions optional LLM integration for natural language queries
• Highlights ability to pass a JSON filter/rename config to any workflow
• Lists key output: a pandas DataFrame of flattened FHIR resources
Main Components & Interactions
a. Data Sources
• Local File System:
– Folder of *.json FHIR Bundle files
– Folder of *.ndjson bulk export files
• Remote Services:
– FHIR Server (REST API, Search endpoint)
– Google BigQuery (FHIR dataset)
• (Optional) LLM Provider (e.g., OpenAI via llama-index)
b. Core Processing Modules
– BaseFhiry: common interfaces and config parsing
– FlattenFhir: transforms a single Bundle/Resource into a flat dict
– Fhirndjson / Fhirsearch / Bqsearch: source‐specific clients calling FlattenFhir
– Parallel: orchestrates multi‐file ingestion and pandas DataFrame assembly
c. Outputs
– pandas DataFrame object
– (Optional) textual representation passed to an LLM
– (Optional) DataFrame passed into ML frameworks (TensorFlow, PyTorch)
d. External Libraries & Frameworks
– pandas, requests, google-cloud-bigquery, llama-index (optional)
Architectural Patterns & Design Principles
• Modular Pipeline: Each ingestion path implements a common interface and shares the flattening logic.
• Single Responsibility: Modules focused on one source type or transformation step.
• Parallel Processing: File‐based ingestion leverages concurrency for speed.
• Configuration‐Driven: JSON‐based filters and renaming control output schema.