How coderxio/sagerx works
Main Components
a. Data Sources Layer:
• External data sources (DailyMed, FDA, RxNorm, Orange Book, etc.)
• UMLS API connection
• AWS S3 and GCP (BigQuery/Cloud Storage) integrations
b. Data Pipeline Layer:
• Apache Airflow orchestration
• Individual DAGs for different data sources (shown in /airflow/dags/)
• Extract and Load processes
c. Transformation Layer:
• dbt transformation pipeline
• Hierarchical structure:
• Staging models (/dbt/sagerx/models/staging/)
• Intermediate models (/dbt/sagerx/models/intermediate/)
• Marts models (/dbt/sagerx/models/marts/)
d. Storage Layer:
• PostgreSQL database
• Google Cloud Storage
• BigQuery
e. Access Layer:
• pgAdmin interface
• API endpoints (if any)
[External Data Sources]
• ↓
[Airflow Orchestration]
• ↓
[Extract & Load Pipeline]
• ↓
[dbt Transformation]
• ↓
[Storage Solutions]
• ↓
[Access Layer]
Important Elements to Highlight
a. Data Flow Patterns:
• Parallel processing of different data sources
• Transformation sequences
• Data validation points
b. Integration Points:
• UMLS API authentication
• Cloud storage connections
• Database interactions
c. Key Components:
• Airflow scheduler
• dbt models hierarchy
• PostgreSQL database
• Cloud storage systems
Additional Details
• Annotations for critical configurations
• Version control integration points
• Scheduled vs. on-demand processes
Specific Implementation Notes
• Environment variable dependencies
• Scaling points
Remember to include version information and note any dependencies between components, especially for the ETL processes and dbt transformations.
This architecture should reflect the modular nature of the system, where new data sources can be added through additional DAGs and corresponding dbt models.