# dermatologist/fhiry architecture

> Architecture diagram of the GitHub repository dermatologist/fhiry, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dermatologist/fhiry
- Repository: https://github.com/dermatologist/fhiry

## Overview

##### Project Type and Purpose
- This is a Python library (package) called **fhiry** that provides a virtual, flattened view of FHIR (Fast Healthcare Interoperability Resources) data in pandas DataFrames for analytics, AI/ML, and LLM‐based queries.
- It supports four primary data sources:
- A folder of FHIR Bundle JSON files
- A folder of NDJSON files (bulk data export)
- A FHIR REST API (FHIR Search)
- Google BigQuery FHIR tables

##### File Structure
- **Top‐level directories**
- `src/fhiry/` – core modules:
– `base_fhiry.py` – common base classes and utilities
– `parallel.py` – orchestrates parallel processing of files
– `fhirndjson.py` – NDJSON ingestion and flattening
– `fhirsearch.py` – FHIR search API client and flattening
– `bqsearch.py` – BigQuery client adapter and flattening
– `flattenfhir.py` – logic to flatten a FHIR Bundle to a tabular form
- `examples/` – usage scripts (e.g., LLM example)
- `notebooks/` – Jupyter notebooks demonstrating workflows
- `tests/` – unit tests and test resources
- `docs/` – Sphinx documentation site
- **Configuration & CI**
- `pyproject.toml`/`setup.py` – packaging and dependencies
- `.github/workflows/` / `tox.ini` / `.devcontainer/` – CI builds, tests, and dev environment

##### README for Additional Insights
- Describes four ingestion workflows (Bundles, NDJSON, FHIR Search, BigQuery)
- Mentions optional LLM integration for natural language queries
- Highlights ability to pass a JSON filter/rename config to any workflow
- Lists key output: a pandas DataFrame of flattened FHIR resources

##### Main Components & Interactions
a. **Data Sources**
- Local File System:
– Folder of `*.json` FHIR Bundle files
– Folder of `*.ndjson` bulk export files
- Remote Services:
– FHIR Server (REST API, Search endpoint)
– Google BigQuery (FHIR dataset)
- (Optional) LLM Provider (e.g., OpenAI via llama-index)
b. **Core Processing Modules**
– **BaseFhiry**: common interfaces and config parsing
– **FlattenFhir**: transforms a single Bundle/Resource into a flat dict
– **Fhirndjson** / **Fhirsearch** / **Bqsearch**: source‐specific clients calling FlattenFhir
– **Parallel**: orchestrates multi‐file ingestion and pandas DataFrame assembly
c. **Outputs**
– pandas DataFrame object
– (Optional) textual representation passed to an LLM
– (Optional) DataFrame passed into ML frameworks (TensorFlow, PyTorch)
d. **External Libraries & Frameworks**
– pandas, requests, google-cloud-bigquery, llama-index (optional)

##### Architectural Patterns & Design Principles
- **Modular Pipeline**: Each ingestion path implements a common interface and shares the flattening logic.
- **Single Responsibility**: Modules focused on one source type or transformation step.
- **Parallel Processing**: File‐based ingestion leverages concurrency for speed.
- **Configuration‐Driven**: JSON‐based filters and renaming control output schema.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        JSONFolder["FHIR Bundle JSON Folder"]:::external
        NDJSONFolder["FHIR NDJSON Folder"]:::external
        FHIRServer["FHIR REST API"]:::external
        BigQuery["Google BigQuery FHIR"]:::external
        LLMProvider["LLM Provider"]:::external
    end

    %% Core Modules
    subgraph "Core Modules"
        BaseFhiry["BaseFhiry"]:::core
        FlattenFhir["FlattenFhir"]:::core
        FhiryMain["fhiry (CLI Entry)"]:::core
    end

    %% Ingestion Clients
    subgraph "Ingestion Clients"
        Fhirndjson["Fhirndjson Client"]:::ingest
        Fhirsearch["Fhirsearch Client"]:::ingest
        Bqsearch["Bqsearch Client"]:::ingest
    end

    %% Orchestration
    Parallel["Parallel Orchestrator"]:::orchestrator

    %% Output
    PDF["pandas DataFrame"]:::output
    Consumer["LLM/Text & ML Frameworks"]:::external

    %% User Examples & Notebooks
    subgraph "User Examples & Notebooks"
        LLMExample["LLM Integration Example"]:::example
        NDJSONNB["NDJSON Notebook"]:::example
        BigQueryNB["BigQuery Notebook"]:::example
        SyntheaNB["Synthea Bundle Notebook"]:::example
    end

    %% Tests
    subgraph "Tests"
        UT1["tests/test_fhiry.py"]:::test
        UT2["tests/test_flattenfhir.py"]:::test
        UT3["tests/test_fhirndjson.py"]:::test
        UT4["tests/test_fhirsearch.py"]:::test
        UT5["tests/test_bqsearch.py"]:::test
    end

    %% CI & Config
    subgraph "CI & Config"
        Config["pyproject.toml & setup.py"]:::external
        CI[".github/workflows"]:::external
        Devcontainer[".devcontainer"]:::external
    end

    %% Connections
    JSONFolder --> Parallel
    NDJSONFolder --> Fhirndjson
    FHIRServer --> Fhirsearch
    BigQuery --> Bqsearch

    Fhirndjson --> FlattenFhir
    Fhirsearch --> FlattenFhir
    Bqsearch --> FlattenFhir

    FlattenFhir --> Parallel
    Parallel --> PDF
    PDF --> Consumer

    FhiryMain --> Parallel
    FhiryMain --> Fhirndjson
    FhiryMain --> Fhirsearch
    FhiryMain --> Bqsearch

    PDF --> LLMExample
    NDJSONNB --> Parallel
    BigQueryNB --> Bqsearch
    SyntheaNB --> FlattenFhir

    UT1 --> BaseFhiry
    UT2 --> FlattenFhir
    UT3 --> Fhirndjson
    UT4 --> Fhirsearch
    UT5 --> Bqsearch

    Config --> BaseFhiry
    CI --> UT1
    Devcontainer --> BaseFhiry

    %% Click Events
    click BaseFhiry "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/base_fhiry.py"
    click FlattenFhir "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/flattenfhir.py"
    click Fhirndjson "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/fhirndjson.py"
    click Fhirsearch "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/fhirsearch.py"
    click Bqsearch "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/bqsearch.py"
    click Parallel "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/parallel.py"
    click FhiryMain "https://github.com/dermatologist/fhiry/blob/develop/src/fhiry/fhiry.py"
    click LLMExample "https://github.com/dermatologist/fhiry/blob/develop/examples/llm_example.py"
    click NDJSONNB "https://github.com/dermatologist/fhiry/blob/develop/notebooks/ndjson.ipynb"
    click BigQueryNB "https://github.com/dermatologist/fhiry/blob/develop/notebooks/gcp.ipynb"
    click SyntheaNB "https://github.com/dermatologist/fhiry/blob/develop/notebooks/synthea.ipynb"
    click UT1 "https://github.com/dermatologist/fhiry/blob/develop/tests/test_fhiry.py"
    click UT2 "https://github.com/dermatologist/fhiry/blob/develop/tests/test_flattenfhir.py"
    click UT3 "https://github.com/dermatologist/fhiry/blob/develop/tests/test_fhirndjson.py"
    click UT4 "https://github.com/dermatologist/fhiry/blob/develop/tests/test_fhirsearch.py"
    click UT5 "https://github.com/dermatologist/fhiry/blob/develop/tests/test_bqsearch.py"

    %% Styles
    classDef core fill:#f9f,stroke:#333,stroke-width:2px
    classDef ingest fill:#9f9,stroke:#333,stroke-width:2px
    classDef orchestrator fill:#ff9,stroke:#333,stroke-width:2px
    classDef external fill:#88f,stroke:#333,stroke-width:2px
    classDef output fill:#ffc,stroke:#333,stroke-width:2px
    classDef example fill:#fcf,stroke:#333,stroke-width:2px
    classDef test fill:#ccc,stroke:#333,stroke-width:2px
```
