# jibinb961/clinical_trial_pipeline architecture

> Architecture diagram of the GitHub repository jibinb961/clinical_trial_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jibinb961/clinical_trial_pipeline
- Repository: https://github.com/jibinb961/clinical_trial_pipeline

## Overview

##### Project type and purpose
- This is a cloud-native, full-stack data pipeline with an orchestration layer (Prefect), a containerized ETL/enrichment/analysis pipeline, and a Streamlit-based visualization UI.
- Its goal is to fetch clinical-trial data, enrich drug information via ChEMBL/Google Gemini, cache results in SQLite on GCS, produce CSV/Parquet/plots/summaries, and surface them in a web dashboard.

##### File structure
- Top-level:
- Dockerfile, docker-compose.yml, deployment.yaml → container build & deployment descriptors
- .github/workflows/ci.yml → CI/CD pipeline
- prefect.yaml → Prefect deployment config
- app/streamlit_app.py → Streamlit UI
- src/pipeline/ → Python modules: flow.py (orchestration), etl.py, enrich.py, analysis.py, utils.py, config.py
- data/ → optional local raw/processed/figures folders (mirrors GCS structure)
- debug/ → adhoc scripts/tests for API and cleanup
- docs/ → design docs and API guides
- tests/ → unit tests for enrichment and ETL

##### README architecture
- CI/CD: GitHub Actions → Google Artifact Registry
- Orchestration: Prefect Server (Render) triggers Cloud Run jobs
- Pipeline in Cloud Run pulls image, runs ETL/enrich/analysis, interacts with:
- ClinicalTrials.gov API
- SQLite drug cache (in GCS)
- ChEMBL API (fallback to Google Gemini)
- Google Cloud Storage for cache and artifacts
- Visualization: Streamlit app reads artifacts from GCS

##### Main components & their interactions
a. CI/CD
- Component: GitHub Repo + GitHub Actions
- Output: Docker image → Artifact Registry
b. Orchestration
- Component: Prefect Server (hosted)
- Triggers & schedules Cloud Run work pool
c. Compute / Pipeline
- Component: Cloud Run Job → Pipeline container
- Inside: flow.py kicks off etl.py → fetch raw data → parse → enrich.py calls drug-cache logic → utils to read/write GCS/SQLite → analysis.py for plots/summaries
– “Fetch & ETL”
– “Drug enrichment”
– “Cache management”
– “Analysis & Output”
d. Storage & Cache
- ClinicalTrials.gov API (external)
- ChEMBL API & Google Gemini (external)
- GCS Bucket:
– Pipeline outputs (CSV, Parquet, plots, summaries)
e. Visualization
- Component: Streamlit app (either in Cloud Run or local Docker Compose)
- Reads artifacts from GCS, serves UI

##### Architectural patterns & design principles
- Containerized microservices (CI/CD, Orchestration, Compute, UI)
- Orchestration via Prefect → decouples scheduling from compute
- Cache-aside pattern: SQLite cache in GCS with invalidation on new enrichments
- Immutable artifacts in GCS (CSV/Parquet/plots) for reproducibility
- Clear separation of concerns: ETL → Enrichment → Analysis → Visualization

- All top-level components (CI/CD, Prefect, Cloud Run Job, GCS, APIs, Streamlit)
- Sub-components inside the pipeline container aligned with src/pipeline modules
- Decision branches in enrichment: “cache hit” vs “cache miss → ChEMBL → Gemini”
- Deployment contexts: GitHub → Artifact Registry → Cloud Run; Docker-Compose → local Streamlit
- Environment variable injection into Prefect/Cloud Run jobs

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD Layer
    CI_CD["CI/CD (GitHub Actions)"]:::ci
    Artifact["Artifact Registry (GAR)"]:::artifact

    CI_CD -->|Build & Push Image| Artifact

    %% Orchestration Layer
    Prefect["Prefect Server"]:::orchestration
    Artifact -->|Pull Image| Prefect

    Prefect -->|Trigger Job| CloudRun

    %% Compute Layer
    subgraph "Cloud Run (Pipeline Job)" 
        direction TB
        Flow["flow.py"]:::compute
        ETL["etl.py"]:::compute
        Enrich["enrich.py"]:::compute
        Gemini["gemini_utils.py"]:::compute
        Utils["utils.py"]:::compute
        Analysis["analysis.py"]:::compute
        Config["config.py"]:::compute

        Flow -->|calls| ETL
        ETL -->|processes raw data| Enrich
        Enrich -->|cache logic| Utils
        Enrich -->|on miss| Gemini
        Utils -->|I/O| Analysis
        Config --- Flow
    end

    Prefect -->|Execute Container| Flow

    %% External APIs
    subgraph "External Services"
        direction TB
        CTgov["ClinicalTrials.gov API"]:::external
        ChEMBL["ChEMBL API"]:::external
        GeminiAPI["Google Gemini"]:::external
    end

    ETL -->|JSON| CTgov
    Enrich -->|Drug Data| ChEMBL
    Gemini -->|Fallback| GeminiAPI

    %% Storage Layer
    subgraph "GCS Bucket" 
        direction TB
        Cache["SQLite Cache"]:::storage
        Outputs["Pipeline Outputs\n(CSV/Parquet/Plots)"]:::storage
    end

    Utils -->|Read/Write| Cache
    Analysis -->|Write Artifacts| Outputs
    ETL -->|Save Raw| Outputs

    %% Visualization Layer
    Streamlit["Streamlit App"]:::visualization
    Outputs -->|Read Artifacts| Streamlit
    LocalDocker["Local Docker-Compose\n(Streamlit)"]:::visualization
    LocalDocker -->|Run UI| Streamlit

    %% Local Data Mirrors
    subgraph "Local Data (mirrors GCS)"
        direction TB
        Raw["data/raw"]:::storage
        Processed["data/processed"]:::storage
        Figures["data/figures"]:::storage
    end

    ETL --> Raw
    Analysis --> Figures
    Analysis --> Processed

    %% Click Events
    click CI_CD "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/.github/workflows/ci.yml"
    click CI_CD "https://github.com/jibinb961/clinical_trial_pipeline/tree/main/Dockerfile"
    click CI_CD "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/deployment.yaml"
    click CI_CD "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/docker-compose.yml"
    click Prefect "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/prefect.yaml"
    click Flow "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/flow.py"
    click ETL "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/etl.py"
    click Enrich "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/enrich.py"
    click Gemini "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/gemini_utils.py"
    click Utils "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/utils.py"
    click Analysis "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/analysis.py"
    click Config "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/src/pipeline/config.py"
    click Raw "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/data/raw/.gitkeep"
    click Processed "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/data/processed/.gitkeep"
    click Figures "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/data/figures/.gitkeep"
    click Streamlit "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/app/streamlit_app.py"
    click LocalDocker "https://github.com/jibinb961/clinical_trial_pipeline/blob/main/docker-compose.yml"

    %% Styles
    classDef ci fill:#cce5ff,stroke:#007bff,color:#004085,stroke-width:2px;
    classDef orchestration fill:#ffe5b4,stroke:#ff8c00,color:#8b4000,stroke-width:2px;
    classDef compute fill:#d4edda,stroke:#28a745,color:#155724,stroke-width:2px;
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#383d41,stroke-width:2px,stroke-dasharray: 5 5;
    classDef artifact fill:#dcdcdc,stroke:#6f42c1,color:#4b0082,stroke-width:2px,stroke-dasharray: 2 2;
    classDef external fill:#f8f9fa,stroke:#adb5bd,color:#6c757d,stroke-width:2px;
    classDef visualization fill:#d1ecf1,stroke:#17a2b8,color:#0c5460,stroke-width:2px;
```
