# coderxio/sagerx architecture

> Architecture diagram of the GitHub repository coderxio/sagerx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/coderxio/sagerx
- Repository: https://github.com/coderxio/sagerx

## Overview

##### Main Components

a. Data Sources Layer:
- External data sources (DailyMed, FDA, RxNorm, Orange Book, etc.)
- UMLS API connection
- AWS S3 and GCP (BigQuery/Cloud Storage) integrations

b. Data Pipeline Layer:
- Apache Airflow orchestration
- Individual DAGs for different data sources (shown in /airflow/dags/)
- Extract and Load processes

c. Transformation Layer:
- dbt transformation pipeline
- Hierarchical structure:
- Staging models (/dbt/sagerx/models/staging/)
- Intermediate models (/dbt/sagerx/models/intermediate/)
- Marts models (/dbt/sagerx/models/marts/)

d. Storage Layer:
- PostgreSQL database
- Google Cloud Storage
- BigQuery

e. Access Layer:
- pgAdmin interface
- API endpoints (if any)

**[External Data Sources]**
- ↓
**[Airflow Orchestration]**
- ↓
**[Extract & Load Pipeline]**
- ↓
**[dbt Transformation]**
- ↓
**[Storage Solutions]**
- ↓
**[Access Layer]**

##### Important Elements to Highlight

a. Data Flow Patterns:
- Parallel processing of different data sources
- Transformation sequences
- Data validation points

b. Integration Points:
- UMLS API authentication
- Cloud storage connections
- Database interactions

c. Key Components:
- Airflow scheduler
- dbt models hierarchy
- PostgreSQL database
- Cloud storage systems

##### Additional Details

- Annotations for critical configurations
- Version control integration points
- Scheduled vs. on-demand processes

##### Specific Implementation Notes

- Environment variable dependencies
- Scaling points

Remember to include version information and note any dependencies between components, especially for the ETL processes and dbt transformations.

This architecture should reflect the modular nature of the system, where new data sources can be added through additional DAGs and corresponding dbt models.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef external fill:#b3d1ff,stroke:#333
    classDef processing fill:#90EE90,stroke:#333
    classDef storage fill:#FFE5B4,stroke:#333
    classDef access fill:#DDA0DD,stroke:#333
    classDef critical fill:#FFB6C6,stroke:#333
    classDef cloud fill:#E6E6E6,stroke:#333

    %% External Data Sources
    subgraph DataSources["External Data Sources"]
        DailyMed["DailyMed"]:::external
        FDA["FDA"]:::external
        RxNorm["RxNorm"]:::external
        OrangeBook["Orange Book"]:::external
        PurpleBook["Purple Book"]:::external
        UMLS["UMLS API"]:::critical
    end

    %% Cloud Services
    subgraph CloudServices["Cloud Services"]
        AWS_S3["AWS S3"]:::cloud
        GCP["Google Cloud Platform"]:::cloud
        BigQuery["BigQuery"]:::cloud
        GCS["Cloud Storage"]:::cloud
    end

    %% Data Pipeline Layer
    subgraph Pipeline["Data Pipeline Layer"]
        Airflow["Apache Airflow"]:::processing
        DailyMedDAG["DailyMed DAG"]:::processing
        FDADAG["FDA DAG"]:::processing
        RxNormDAG["RxNorm DAG"]:::processing
        OrangeDAG["Orange Book DAG"]:::processing
        PurpleDAG["Purple Book DAG"]:::processing
    end

    %% Transformation Layer
    subgraph Transform["dbt Transformation"]
        direction TB
        Staging["Staging Models"]:::processing
        Intermediate["Intermediate Models"]:::processing
        Marts["Marts Models"]:::processing
    end

    %% Storage Layer
    subgraph Storage["Storage Layer"]
        PostgreSQL["PostgreSQL"]:::storage
    end

    %% Access Layer
    subgraph Access["Access Layer"]
        PGAdmin["pgAdmin Interface"]:::access
        APIEndpoints["API Endpoints"]:::access
    end

    %% Connections
    DailyMed --> DailyMedDAG
    FDA --> FDADAG
    RxNorm --> RxNormDAG
    OrangeBook --> OrangeDAG
    PurpleBook --> PurpleDAG
    UMLS --> Pipeline

    DailyMedDAG & FDADAG & RxNormDAG & OrangeDAG & PurpleDAG --> Staging
    Staging --> Intermediate
    Intermediate --> Marts
    Marts --> PostgreSQL
    PostgreSQL --> PGAdmin
    PostgreSQL --> APIEndpoints

    Pipeline --> AWS_S3
    Pipeline --> GCP
    GCP --> BigQuery
    GCP --> GCS

    %% Click Events
    click Airflow "https://github.com/coderxio/sagerx/tree/main/airflow/"
    click DailyMedDAG "https://github.com/coderxio/sagerx/tree/main/airflow/dags/dailymed/"
    click FDADAG "https://github.com/coderxio/sagerx/tree/main/airflow/dags/fda_ndc/"
    click RxNormDAG "https://github.com/coderxio/sagerx/tree/main/airflow/dags/rxnorm/"
    click OrangeDAG "https://github.com/coderxio/sagerx/tree/main/airflow/dags/orange_book/"
    click PurpleDAG "https://github.com/coderxio/sagerx/tree/main/airflow/dags/purple_book/"
    click Staging "https://github.com/coderxio/sagerx/tree/main/dbt/sagerx/models/staging/"
    click Intermediate "https://github.com/coderxio/sagerx/tree/main/dbt/sagerx/models/intermediate/"
    click Marts "https://github.com/coderxio/sagerx/tree/main/dbt/sagerx/models/marts/"
    click PostgreSQL "https://github.com/coderxio/sagerx/tree/main/postgres/"
    click PGAdmin "https://github.com/coderxio/sagerx/tree/main/pgadmin/"

    %% Legend
    classDef legend fill:#f9f9f9,stroke:#333
    subgraph Legend["Legend"]
        L1["External Sources"]:::external
        L2["Processing"]:::processing
        L3["Storage"]:::storage
        L4["Access Layer"]:::access
        L5["Critical Systems"]:::critical
        L6["Cloud Services"]:::cloud
    end
```
