How broadinstitute/gene-trends works
Project type and purpose
• This is a full-stack, data-centric web application called “Gene Trends.”
• Purpose: ingest biological citation and page-view data for various genes, process that data with a Python pipeline, produce processed TSV data and static assets, and present those trends via a web UI (both a React SPA and a plain-JS fallback).
File structure
• Top-level directories:
• data/ ← Raw input TSVs (PubMed citations, Wikipedia views, etc.)
• gene_files/ ← Per-gene extracted files (original and date-sorted)
• gene_trends/← Python package: modules for citations, views, merging, utility libraries
• processed_tsv/ ← Output of the Python ETL pipeline
• ui/ ← Two frontend flavors:
– ui/app: React TypeScript app with build config, components, public data
– ui/plain: Vanilla JS + HTML version under plain/js, plain/data
• docs/ ← Static build output served (via GitHub Pages)
• deploy.sh ← Script to build and copy UI + possibly run Python generator into docs
• requirements*.txt, tests/, test/ ← Python dependencies and test suites
README for additional insights
• Describes how to install and run the React app locally (npm install/run).
• Shows how to set up a Python venv and run the main pipeline script (gene_trends/gene_trends.py).
• Deploys everything with deploy.sh into docs/ for GitHub Pages.
Main components and interactions
a. Raw Data Store
• data/*.tsv, organisms.tsv → source citation and view data
b. ETL & Processing Layer (Python)
• gene_trends/citations: citation counts, enrichment, date extraction
• gene_trends/views: page-view aggregation, gene-page mapping
• gene_trends/merge_trends.py: combine citation + view trends
• gene_trends/gene_trends.py: orchestrates full pipeline, writes processed_tsv/ and gene_files_datesorted/
c. Processed Data Store
• processed_tsv/*.tsv ← cleaned, merged trend data ready for consumption
• gene_files_datesorted/*.tsv ← per-gene files sorted by date
d. Frontend Build & Serving
• ui/app (React) build → static JS/CSS under docs/static/js, docs/index.html
• ui/plain (vanilla) → plain HTML + JS under docs/static (if used)
• deploy.sh: runs Python pipeline, builds React app, copies outputs into docs/
• docs/: artifact hosting root for GitHub Pages
e. End-User Interaction
• Browser → fetch docs/index.html + static assets → interactive TrendsDashboard
Architectural patterns and design principles
• ETL pipeline (Extract → Transform → Load) in Python
• Static site generation for UI (React build + fallback plain JS)
• Clear separation: Data layer, Processing layer, Presentation layer
• Modular design: each gene-processing step is in its own Python module under gene_trends/
• Data ingestion sources (data/*)
• ETL modules (gene_trends/citations, gene_trends/views, merge_trends)
• Data outputs (processed_tsv/, gene_files_datesorted/)
• Frontend projects (ui/app vs. ui/plain) and their build pipelines
• Deployment target (docs/, GitHub Pages)
• User browser as the end point
• Dataflow: raw → process → output → build → serve → client
• Arrowheads indicate direction of data or control flow
• Labels for key technologies/frameworks at each step