# broadinstitute/gene-trends architecture

> Architecture diagram of the GitHub repository broadinstitute/gene-trends, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/broadinstitute/gene-trends
- Repository: https://github.com/broadinstitute/gene-trends

## Overview

##### Project type and purpose
- This is a full-stack, data-centric web application called “Gene Trends.”
- Purpose: ingest biological citation and page-view data for various genes, process that data with a Python pipeline, produce processed TSV data and static assets, and present those trends via a web UI (both a React SPA and a plain-JS fallback).

##### File structure
- Top-level directories:
- data/       ← Raw input TSVs (PubMed citations, Wikipedia views, etc.)
- gene_files/ ← Per-gene extracted files (original and date-sorted)
- gene_trends/← Python package: modules for citations, views, merging, utility libraries
- processed_tsv/ ← Output of the Python ETL pipeline
- ui/         ← Two frontend flavors:
– ui/app: React TypeScript app with build config, components, public data
– ui/plain: Vanilla JS + HTML version under plain/js, plain/data
- docs/       ← Static build output served (via GitHub Pages)
- deploy.sh   ← Script to build and copy UI + possibly run Python generator into docs
- requirements*.txt, tests/, test/ ← Python dependencies and test suites

##### README for additional insights
- Describes how to install and run the React app locally (npm install/run).
- Shows how to set up a Python venv and run the main pipeline script (gene_trends/gene_trends.py).
- Deploys everything with deploy.sh into docs/ for GitHub Pages.

##### Main components and interactions
a. Raw Data Store
- data/*.tsv, organisms.tsv → source citation and view data
b. ETL & Processing Layer (Python)
- gene_trends/citations: citation counts, enrichment, date extraction
- gene_trends/views: page-view aggregation, gene-page mapping
- gene_trends/merge_trends.py: combine citation + view trends
- gene_trends/gene_trends.py: orchestrates full pipeline, writes processed_tsv/ and gene_files_datesorted/
c. Processed Data Store
- processed_tsv/*.tsv ← cleaned, merged trend data ready for consumption
- gene_files_datesorted/*.tsv ← per-gene files sorted by date
d. Frontend Build & Serving
- ui/app (React) build → static JS/CSS under docs/static/js, docs/index.html
- ui/plain (vanilla) → plain HTML + JS under docs/static (if used)
- deploy.sh: runs Python pipeline, builds React app, copies outputs into docs/
- docs/: artifact hosting root for GitHub Pages
e. End-User Interaction
- Browser → fetch docs/index.html + static assets → interactive TrendsDashboard

##### Architectural patterns and design principles
- ETL pipeline (Extract → Transform → Load) in Python
- Static site generation for UI (React build + fallback plain JS)
- Clear separation: Data layer, Processing layer, Presentation layer
- Modular design: each gene-processing step is in its own Python module under gene_trends/

- Data ingestion sources (data/*)
- ETL modules (gene_trends/citations, gene_trends/views, merge_trends)
- Data outputs (processed_tsv/, gene_files_datesorted/)
- Frontend projects (ui/app vs. ui/plain) and their build pipelines
- Deployment target (docs/, GitHub Pages)
- User browser as the end point
- Dataflow: raw → process → output → build → serve → client
- Arrowheads indicate direction of data or control flow
- Labels for key technologies/frameworks at each step

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Raw Data Store"
        RawData["Raw Data Store\ndata/*.tsv\norganisms.tsv"]:::dataStore
    end

    subgraph "ETL & Processing Layer"
        Orchestrator["Pipeline Orchestrator\ngene_trends/gene_trends.py"]:::compute
        Citations["Citations Module\ngene_trends/citations/"]:::compute
        Views["Views Module\ngene_trends/views/"]:::compute
        Merge["Merge Trends Script\ngene_trends/merge_trends.py"]:::compute
        UtilLib["Utility Libraries\ngene_trends/lib.py\netc/gene_categorizations.py"]:::compute
    end

    subgraph "Processed Data Store"
        Processed["Processed TSVs\nprocessed_tsv/"]:::dataStore
        GeneFiles["Per-Gene Sorted Files\ngene_files_datesorted/"]:::dataStore
    end

    subgraph "Frontend Build & Serving"
        ReactBuild["React SPA Build\nui/app/"]:::frontend
        PlainUI["Vanilla JS UI\nui/plain/"]:::frontend
    end

    subgraph "Static Host & Deployment"
        Deploy["Deployment Script\ndeploy.sh"]:::deploy
        subgraph "Static Site"
            Index["Index.html"]:::deploy
            StaticAssets["Static Assets\nstatic/"]:::deploy
        end
    end

    Browser["Browser Client"]:::frontend

    %% Data Flow
    RawData -->|"reads"| Orchestrator
    Orchestrator -->|"uses"| Citations
    Orchestrator -->|"uses"| Views
    Orchestrator -->|"uses"| Merge
    Orchestrator -->|"imports"| UtilLib
    Orchestrator -->|"writes"| Processed
    Orchestrator -->|"writes"| GeneFiles

    Processed -->|"consumed by"| ReactBuild
    Processed -->|"consumed by"| PlainUI
    GeneFiles -->|"optional data"| ReactBuild

    ReactBuild -->|"outputs"| Index
    ReactBuild -->|"outputs"| StaticAssets
    PlainUI -->|"outputs"| StaticAssets

    Deploy -->|"runs ETL"| Orchestrator
    Deploy -->|"builds SPA"| ReactBuild
    Deploy -->|"builds fallback"| PlainUI
    Deploy -->|"copies assets"| Index
    Deploy -->|"copies assets"| StaticAssets

    Index -->|"served to"| Browser
    StaticAssets -->|"served to"| Browser
    Browser -->|"requests"| Index
    Browser -->|"requests"| StaticAssets

    %% Click Events
    click RawData "https://github.com/broadinstitute/gene-trends/tree/main/data/"
    click Citations "https://github.com/broadinstitute/gene-trends/tree/main/gene_trends/citations/"
    click Views "https://github.com/broadinstitute/gene-trends/tree/main/gene_trends/views/"
    click Merge "https://github.com/broadinstitute/gene-trends/blob/main/gene_trends/merge_trends.py"
    click Orchestrator "https://github.com/broadinstitute/gene-trends/blob/main/gene_trends/gene_trends.py"
    click UtilLib "https://github.com/broadinstitute/gene-trends/blob/main/gene_trends/lib.py"
    click UtilLib "https://github.com/broadinstitute/gene-trends/blob/main/gene_trends/etc/gene_categorizations.py"
    click Processed "https://github.com/broadinstitute/gene-trends/tree/main/processed_tsv/"
    click GeneFiles "https://github.com/broadinstitute/gene-trends/tree/main/gene_files/gene_files_datesorted/"
    click ReactBuild "https://github.com/broadinstitute/gene-trends/tree/main/ui/app/"
    click PlainUI "https://github.com/broadinstitute/gene-trends/tree/main/ui/plain/"
    click Deploy "https://github.com/broadinstitute/gene-trends/blob/main/deploy.sh"
    click Index "https://github.com/broadinstitute/gene-trends/blob/main/docs/index.html"
    click StaticAssets "https://github.com/broadinstitute/gene-trends/tree/main/docs/static/"

    %% Styles
    classDef dataStore fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#000,stroke-width:1px
    classDef frontend fill:#FFA500,stroke:#000,stroke-width:1px
    classDef deploy fill:#D3D3D3,stroke:#000,stroke-width:1px
```
