How chufangao/ctod works
Project Type and Purpose
• CTO is a research‐oriented data‐processing pipeline that ingests multiple external data sources (clinical trial registries, PubMed abstracts, news headlines, stock prices, FDA approvals) and applies weakly supervised labeling functions (LFs) to generate a large‐scale benchmark dataset of clinical trial outcomes.
• It is not a typical frontend/backend web app but a modular pipeline of Python scripts, notebooks, and model components orchestrated by shell scripts (e.g., pipeline.sh).
File Structure
• Top‐level orchestrator scripts: pipeline.sh (runs full pipeline), update_news.sh, update_labels.sh, arrange_labels.py, download_ctti.py.
• Module directories (each responsible for a data source or functionality):
• clinical_trial_linkage/ – linking trials across phases and to FDA approvals
• llm_prediction_on_pubmed/ – retrieving PubMed abstracts and prompting GPT‐3.5 for outcome predictions
• news_headlines/ – scraping Google News via SerpAPI and classifying sentiment with FinBERT
• stock_price/ – downloading Yahoo Finance data and computing moving‐average slopes
• labeling/ – combining all LFs, threshold tuning, and final Random Forest aggregator
• Supporting directories: baselines/ (baseline models and scripts), tutorials/ (Jupyter notebooks and Colab examples), requirements.txt, CI workflows (.github/workflows/static.yml).
Extract Insights from the README
• Each module is a self‐contained “weak supervision” source with its own extraction, preprocessing, scoring, and labeling scripts.
• Final label synthesis uses a Random Forest over all LF outputs.
• External dependencies: OpenAI API for GPT‐3.5, PubMed/BERT models, SerpAPI, FinBERT, HuggingFace dataset release.
Main Components and Data Flows
a. Data Sources (external services):
• ClinicalTrials.gov API and local trial datasets
• PubMed API for abstracts
• FDA approvals files
• SerpAPI for news headlines
• Yahoo Finance for historical stock prices
b. Ingestion & Preprocessing Modules:
• clinical_trial_linkage/download_data.py → process_drugbank.py → match_fda_approvals.py → create_trial_linkage.py
• llm_prediction_on_pubmed/extract_pubmed_abstracts*.py → get_llm_predictions.py
• news_headlines/get_news*.py → support_functions.py
• stock_price/get_stocks.py → scrape_amendments.py → slope_calculation.ipynb
c. Feature Extraction & Labeling Functions (LFs):
• For each module, compute weak label scores (text‐similarity reranking, GPT‐3.5 classification, sentiment polarity, SMA slope)
• labeling/lfs.py merges these scores, applies quantile thresholds
d. Label Aggregation:
• labeling/train_random_forest.py (inside lfs.py) trains a Random Forest on LF outputs to produce final labels
e. Baselines & Evaluation:
• baselines/baselines.py and supporting scripts compare LF‐based labels against supervised baselines
f. User Interface & Distribution:
• tutorials/ notebooks for interactive exploration
• pipeline.sh for end‐to‐end runs
• release of dataset to HuggingFace and project website
Architectural Patterns and Design Principles
• Modular separation by data source (each weak supervision source is a separate sub‐pipeline)
• Pipeline orchestration via shell scripts and reproducible notebooks
• Clear separation between data ingestion, feature extraction, labeling, and evaluation
• Use of external APIs and models encapsulated behind thin Python wrappers