# chufangao/ctod architecture

> Architecture diagram of the GitHub repository chufangao/ctod, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chufangao/ctod
- Repository: https://github.com/chufangao/ctod

## Overview

##### Project Type and Purpose
- CTO is a research‐oriented data‐processing pipeline that ingests multiple external data sources (clinical trial registries, PubMed abstracts, news headlines, stock prices, FDA approvals) and applies weakly supervised labeling functions (LFs) to generate a large‐scale benchmark dataset of clinical trial outcomes.
- It is not a typical frontend/backend web app but a modular pipeline of Python scripts, notebooks, and model components orchestrated by shell scripts (e.g., pipeline.sh).

##### File Structure
- Top‐level orchestrator scripts: pipeline.sh (runs full pipeline), update_news.sh, update_labels.sh, arrange_labels.py, download_ctti.py.
- Module directories (each responsible for a data source or functionality):
- clinical_trial_linkage/ – linking trials across phases and to FDA approvals
- llm_prediction_on_pubmed/ – retrieving PubMed abstracts and prompting GPT‐3.5 for outcome predictions
- news_headlines/ – scraping Google News via SerpAPI and classifying sentiment with FinBERT
- stock_price/ – downloading Yahoo Finance data and computing moving‐average slopes
- labeling/ – combining all LFs, threshold tuning, and final Random Forest aggregator
- Supporting directories: baselines/ (baseline models and scripts), tutorials/ (Jupyter notebooks and Colab examples), requirements.txt, CI workflows (.github/workflows/static.yml).

##### Extract Insights from the README
- Each module is a self‐contained “weak supervision” source with its own extraction, preprocessing, scoring, and labeling scripts.
- Final label synthesis uses a Random Forest over all LF outputs.
- External dependencies: OpenAI API for GPT‐3.5, PubMed/BERT models, SerpAPI, FinBERT, HuggingFace dataset release.

##### Main Components and Data Flows
a. Data Sources (external services):
- ClinicalTrials.gov API and local trial datasets
- PubMed API for abstracts
- FDA approvals files
- SerpAPI for news headlines
- Yahoo Finance for historical stock prices
b. Ingestion & Preprocessing Modules:
- clinical_trial_linkage/download_data.py → process_drugbank.py → match_fda_approvals.py → create_trial_linkage.py
- llm_prediction_on_pubmed/extract_pubmed_abstracts*.py → get_llm_predictions.py
- news_headlines/get_news*.py → support_functions.py
- stock_price/get_stocks.py → scrape_amendments.py → slope_calculation.ipynb
c. Feature Extraction & Labeling Functions (LFs):
- For each module, compute weak label scores (text‐similarity reranking, GPT‐3.5 classification, sentiment polarity, SMA slope)
- labeling/lfs.py merges these scores, applies quantile thresholds
d. Label Aggregation:
- labeling/train_random_forest.py (inside lfs.py) trains a Random Forest on LF outputs to produce final labels
e. Baselines & Evaluation:
- baselines/baselines.py and supporting scripts compare LF‐based labels against supervised baselines
f. User Interface & Distribution:
- tutorials/ notebooks for interactive exploration
- pipeline.sh for end‐to‐end runs
- release of dataset to HuggingFace and project website

##### Architectural Patterns and Design Principles
- Modular separation by data source (each weak supervision source is a separate sub‐pipeline)
- Pipeline orchestration via shell scripts and reproducible notebooks
- Clear separation between data ingestion, feature extraction, labeling, and evaluation
- Use of external APIs and models encapsulated behind thin Python wrappers

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    CTgovAPI["ClinicalTrials.gov API"]:::external
    PubMedAPI["PubMed API"]:::external
    FDAfiles["FDA Approvals Files"]:::db
    SerpAPI["SerpAPI (News)"]:::external
    YahooFinance["Yahoo Finance"]:::external
    OpenAIAPI["OpenAI API (GPT-3.5)"]:::external

    %% Ingestion & Preprocessing Modules
    subgraph "Ingestion & Preprocessing" 
        direction TB
        CT_Linkage_Ingest["Clinical Trial Linkage Ingest"]:::ingestion
        PubMed_Ingest["PubMed Abstracts Ingest"]:::ingestion
        News_Ingest["News Headlines Ingest"]:::ingestion
        Stock_Ingest["Stock Price Ingest"]:::ingestion
    end

    %% Feature Extraction & Labeling Functions
    subgraph "Weak Labeling Functions" 
        direction TB
        CT_Linkage_LF["Linkage LF Module"]:::lf
        LLM_LF["LLM Predictions LF Module"]:::lf
        News_LF["News Sentiment LF Module"]:::lf
        Stock_LF["Stock Slope LF Module"]:::lf
    end

    %% Label Aggregation
    Label_Aggregation["Random Forest Aggregator"]:::agg

    %% Baselines & Evaluation
    Baselines["Baselines & Evaluation"]:::agg

    %% Tutorials & Notebooks
    subgraph "Tutorials & Notebooks" 
        direction TB
        Tutorial1["example_collab.ipynb"]:::notebook
        Tutorial2["full_experiments.ipynb"]:::notebook
        Tutorial3["getting_started_cto_vs_top.ipynb"]:::notebook
        Tutorial4["predictions_on_current_trials.ipynb"]:::notebook
        VisUtils["vis_utils.py"]:::notebook
    end

    %% Orchestration Scripts
    subgraph "Orchestration & CI" 
        direction TB
        Pipeline["pipeline.sh"]:::orchestration
        UpdateNews["update_news.sh"]:::orchestration
        UpdateLabels["update_labels.sh"]:::orchestration
        ArrangeLabels["arrange_labels.py"]:::orchestration
        DownloadCTTI["download_ctti.py"]:::orchestration
        CI[".github/workflows/static.yml"]:::orchestration
    end

    %% Data Flows
    CTgovAPI --> CT_Linkage_Ingest --> CT_Linkage_LF --> Label_Aggregation
    FDAfiles --> CT_Linkage_Ingest
    PubMedAPI --> PubMed_Ingest --> LLM_LF --> Label_Aggregation
    OpenAIAPI --> LLM_LF
    SerpAPI --> News_Ingest --> News_LF --> Label_Aggregation
    YahooFinance --> Stock_Ingest --> Stock_LF --> Label_Aggregation

    Label_Aggregation --> Baselines
    Label_Aggregation --> Tutorial1
    Label_Aggregation --> Tutorial2
    Label_Aggregation --> Tutorial3
    Label_Aggregation --> Tutorial4

    %% Orchestration flows
    Pipeline --> CT_Linkage_Ingest
    Pipeline --> PubMed_Ingest
    Pipeline --> News_Ingest
    Pipeline --> Stock_Ingest
    Pipeline --> Label_Aggregation

    UpdateNews --> News_Ingest
    UpdateLabels --> Label_Aggregation
    ArrangeLabels --> Label_Aggregation
    DownloadCTTI --> CT_Linkage_Ingest

    CI --> Pipeline

    %% Click Events
    click Pipeline "https://github.com/chufangao/ctod/blob/main/pipeline.sh"
    click UpdateNews "https://github.com/chufangao/ctod/blob/main/update_news.sh"
    click UpdateLabels "https://github.com/chufangao/ctod/blob/main/update_labels.sh"
    click ArrangeLabels "https://github.com/chufangao/ctod/blob/main/arrange_labels.py"
    click DownloadCTTI "https://github.com/chufangao/ctod/blob/main/download_ctti.py"
    click CT_Linkage_Ingest "https://github.com/chufangao/ctod/tree/main/clinical_trial_linkage/"
    click PubMed_Ingest "https://github.com/chufangao/ctod/tree/main/llm_prediction_on_pubmed/"
    click News_Ingest "https://github.com/chufangao/ctod/tree/main/news_headlines/"
    click Stock_Ingest "https://github.com/chufangao/ctod/tree/main/stock_price/"
    click Label_Aggregation "https://github.com/chufangao/ctod/blob/main/labeling/lfs.py"
    click Tutorial1 "https://github.com/chufangao/ctod/blob/main/tutorials/example_collab.ipynb"
    click Tutorial2 "https://github.com/chufangao/ctod/blob/main/tutorials/full_experiments.ipynb"
    click Tutorial3 "https://github.com/chufangao/ctod/blob/main/tutorials/getting_started_cto_vs_top.ipynb"
    click Tutorial4 "https://github.com/chufangao/ctod/blob/main/tutorials/predictions_on_current_trials.ipynb"
    click VisUtils "https://github.com/chufangao/ctod/blob/main/tutorials/vis_utils.py"
    click CI "https://github.com/chufangao/ctod/blob/main/.github/workflows/static.yml"
    click Baselines "https://github.com/chufangao/ctod/blob/main/baselines/baselines.py"

    %% Styles
    classDef external fill:#E8F1FD,stroke:#0366d6,color:#0366d6,stroke-width:1px;
    classDef db fill:#F3E2A9,stroke:#B8860B,color:#B8860B,stroke-width:1px;
    classDef ingestion fill:#D0EBD0,stroke:#2E8B57,color:#2E8B57,stroke-width:1px;
    classDef lf fill:#D0E7F8,stroke:#1E90FF,color:#1E90FF,stroke-width:1px;
    classDef agg fill:#FDEBD0,stroke:#D35400,color:#D35400,stroke-width:1px;
    classDef orchestration fill:#E5E8E8,stroke:#5D6D7E,color:#5D6D7E,stroke-width:1px,stroke-dasharray: 5 5;
    classDef notebook fill:#F5EEF8,stroke:#8E44AD,color:#8E44AD,stroke-width:1px;
```
