How jibinb961/clinical_trial_pipeline works
Project type and purpose
• This is a cloud-native, full-stack data pipeline with an orchestration layer (Prefect), a containerized ETL/enrichment/analysis pipeline, and a Streamlit-based visualization UI.
• Its goal is to fetch clinical-trial data, enrich drug information via ChEMBL/Google Gemini, cache results in SQLite on GCS, produce CSV/Parquet/plots/summaries, and surface them in a web dashboard.
File structure
• Top-level:
• Dockerfile, docker-compose.yml, deployment.yaml → container build & deployment descriptors
• .github/workflows/ci.yml → CI/CD pipeline
• prefect.yaml → Prefect deployment config
• app/streamlit_app.py → Streamlit UI
• src/pipeline/ → Python modules: flow.py (orchestration), etl.py, enrich.py, analysis.py, utils.py, config.py
• data/ → optional local raw/processed/figures folders (mirrors GCS structure)
• debug/ → adhoc scripts/tests for API and cleanup
• docs/ → design docs and API guides
• tests/ → unit tests for enrichment and ETL
README architecture
• CI/CD: GitHub Actions → Google Artifact Registry
• Orchestration: Prefect Server (Render) triggers Cloud Run jobs
• Pipeline in Cloud Run pulls image, runs ETL/enrich/analysis, interacts with:
• ClinicalTrials.gov API
• SQLite drug cache (in GCS)
• ChEMBL API (fallback to Google Gemini)
• Google Cloud Storage for cache and artifacts
• Visualization: Streamlit app reads artifacts from GCS
Main components & their interactions
a. CI/CD
• Component: GitHub Repo + GitHub Actions
• Output: Docker image → Artifact Registry
b. Orchestration
• Component: Prefect Server (hosted)
• Triggers & schedules Cloud Run work pool
c. Compute / Pipeline
• Component: Cloud Run Job → Pipeline container
• Inside: flow.py kicks off etl.py → fetch raw data → parse → enrich.py calls drug-cache logic → utils to read/write GCS/SQLite → analysis.py for plots/summaries
– “Fetch & ETL”
– “Drug enrichment”
– “Cache management”
– “Analysis & Output”
d. Storage & Cache
• ClinicalTrials.gov API (external)
• ChEMBL API & Google Gemini (external)
• GCS Bucket:
– Pipeline outputs (CSV, Parquet, plots, summaries)
e. Visualization
• Component: Streamlit app (either in Cloud Run or local Docker Compose)
• Reads artifacts from GCS, serves UI
Architectural patterns & design principles
• Containerized microservices (CI/CD, Orchestration, Compute, UI)
• Orchestration via Prefect → decouples scheduling from compute
• Cache-aside pattern: SQLite cache in GCS with invalidation on new enrichments
• Immutable artifacts in GCS (CSV/Parquet/plots) for reproducibility
• Clear separation of concerns: ETL → Enrichment → Analysis → Visualization
• All top-level components (CI/CD, Prefect, Cloud Run Job, GCS, APIs, Streamlit)
• Sub-components inside the pipeline container aligned with src/pipeline modules
• Decision branches in enrichment: “cache hit” vs “cache miss → ChEMBL → Gemini”
• Deployment contexts: GitHub → Artifact Registry → Cloud Run; Docker-Compose → local Streamlit
• Environment variable injection into Prefect/Cloud Run jobs