# alleninstitute/micronsbinder architecture

> Architecture diagram of the GitHub repository alleninstitute/micronsbinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alleninstitute/micronsbinder
- Repository: https://github.com/alleninstitute/micronsbinder

## Overview

##### Project type and purpose
- This is a teaching‐tool repository (“MicronsBinder”) comprised of Jupyter notebooks for interactive analysis of large‐scale connectomics data (EM reconstructions and two‐photon calcium imaging).
- It is designed to launch on mybinder.org (via a Docker container) or locally via a conda environment. Users explore data through notebooks that pull from cloud services, run analyses, and produce figures.

##### File structure
- Top‐level config:
- .gitignore, .travis.yml, LICENSE, CONTRIBUTING.md
- environment.yml, apt.txt (binder requirements)
- postBuild (runs after binder build)
- Makefile (data‐download orchestration)
- notebooks/ directory broken into:
- intro/ (Phase 1 demo notebooks)
- mm3_intro/ (Phase 3 introductory notebooks with CAVE setup, mesh access, synapse queries)
- vignette_analysis/ (deeper analyses in subfolders function, mitochondria, motifs, resource_statistics)
- Within each vignette_analysis subfolder:
- lib/ (shared Python modules)
- scripts/ (data‐prep and compute scripts)
- assets/ or figures/ (output)

##### README for additional insights
- Describes the binder build process, Python 3.7+ conda environment, and how notebooks connect to:
- CAVE services (Connectome Annotation Versioning Engine)
- CloudVolume (volumetric data, meshes, skeletons)
- MeshParty (mesh download, analysis, visualization via VTK)
- Neuroglancer (3D web visualization)
- DashDataFrame/Dash for interactive plots

##### Main components
a. Binder build environment
- Dockerfile (implied by binder), apt.txt → apt packages
- environment.yml → conda environment
- postBuild script
b. JupyterLab front‐end
- Notebook UI where users launch and run .ipynb files
c. Notebook code
- notebooks/intro, mm3_intro, vignette_analysis/*
- lib/ modules and scripts for data processing
- Makefile targets for bulk data download
d. Data access layer
- CAVE REST API (annotation/versioning)
- CloudVolume API (EM images, segmentation, meshes, skeletons)
e. Analysis & visualization libraries
- MeshParty, trimesh, scipy, vtk
- neuroglancer (JS viewer), DashDataFrame / Dash
- pandas, matplotlib, plotly
f. External services & storage
- AWS/GCP buckets or cloud storage hosting micrograph volumes and segmentation
- Binder service (mybinder.org) or local conda environment

##### Component relationships & data flows
- Binder build → Docker container → conda env → JupyterLab
- User opens a notebook → notebook loads Python libs → calls CAVE & CloudVolume → fetches raw/segmented volumes, meshes, skeletons
- Data flows into lib/ scripts for processing (e.g. compute_degree.py) → outputs saved under assets/ or figures/
- Visualization routines call neuroglancer (link generation) or render locally with VTK/Dash
- Makefile orchestrates bulk data downloads into notebook subfolders

##### Architectural patterns & design principles
- Separation of concerns:
- Environment provisioning (environment.yml, apt.txt, postBuild)
- Interactive layer (Jupyter notebooks)
- Data‐access layer (CloudVolume, CAVE)
- Analysis modules (lib/), scripts (compute_*.py)
- Output storage (assets/, figures/)
- Declarative orchestration:
- Makefile targets for reproducible data fetch
- Modular code:
- Python packages in lib/ usable by multiple notebooks
- Containerized reproducibility:
- Binder build ensures identical Python environment on cloud and local

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment Provisioning
    subgraph "Environment Provisioning"
        EnvGitignore[".gitignore"]:::config
        EnvTravis[".travis.yml"]:::config
        EnvContrib["CONTRIBUTING.md"]:::config
        EnvLicense["LICENSE"]:::config
        EnvReadme["README.md"]:::config
        EnvEnvYml["environment.yml"]:::envfile
        EnvApt["apt.txt"]:::envfile
        EnvPost["postBuild"]:::envfile
    end

    Makefile["Makefile"]:::orchestration
    Docker["Docker Container"]:::env
    JupyterUI["JupyterLab Front-End"]:::ui

    %% Notebooks
    subgraph "Notebooks"
        subgraph "Phase 1 Intro"
            IntroNB["notebooks/intro/"]:::notebooks
        end
        subgraph "Phase 3 mm3_intro"
            MM3Dir["notebooks/mm3_intro/"]:::notebooks
            CAVEsetup["CAVEsetup.ipynb"]:::notebook
            MeshAccess["MeshAccess.ipynb"]:::notebook
            SynapseQuery["SynapseAndAnnotationQuery.ipynb"]:::notebook
            VisualSyn["VisualizingSynapsesWithMeshes.ipynb"]:::notebook
        end
        subgraph "Vignette Analyses"
            subgraph "Function Module"
                FuncLib["notebooks/vignette_analysis/function/lib/"]:::code
                FuncScripts["notebooks/vignette_analysis/function/scripts/"]:::code
            end
            subgraph "Mitochondria Module"
                MitoLib["notebooks/vignette_analysis/mitochondria/lib/"]:::code
                MitoScripts["notebooks/vignette_analysis/mitochondria/scripts/"]:::code
            end
            subgraph "Motifs Module"
                MotifsUtils["notebooks/vignette_analysis/motifs/utils/"]:::code
            end
            subgraph "Resource Statistics Module"
                ResLib["notebooks/vignette_analysis/resource_statistics/lib/"]:::code
                ResScripts["notebooks/vignette_analysis/resource_statistics/scripts/"]:::code
            end
        end
    end

    %% Data Access Layer
    subgraph "Data Access Layer"
        CAVE["CAVE REST API"]:::dataaccess
        CloudVol["CloudVolume API"]:::dataaccess
        Storage["AWS/GCP Cloud Storage"]:::external
    end

    %% Analysis & Visualization Libraries
    subgraph "Analysis Libraries"
        MeshParty["MeshParty"]:::analysis
        SciPy["scipy, pandas"]:::analysis
        Trimesh["trimesh, vtk"]:::analysis
    end
    subgraph "Visualization Tools"
        Neuroglancer["Neuroglancer (JS Viewer)"]:::viz
        Dash["DashDataFrame / Dash"]:::viz
    end

    BinderService["mybinder.org"]:::external

    %% Relationships
    EnvEnvYml -->|defines| Docker
    EnvApt -->|provides| Docker
    EnvPost -->|configures| Docker
    Makefile -->|orchestrates data download| IntroNB
    Makefile -->|orchestrates data download| FuncScripts
    Docker -->|launches| JupyterUI
    JupyterUI -->|opens| IntroNB
    JupyterUI -->|opens| MM3Dir
    MM3Dir -->|contains| CAVEsetup
    MM3Dir -->|contains| MeshAccess
    MM3Dir -->|contains| SynapseQuery
    MM3Dir -->|contains| VisualSyn
    JupyterUI -->|runs code| FuncLib
    JupyterUI -->|runs scripts| FuncScripts
    JupyterUI -->|runs code| MitoLib
    JupyterUI -->|runs scripts| MitoScripts
    JupyterUI -->|runs code| MotifsUtils
    JupyterUI -->|runs code| ResLib
    JupyterUI -->|runs scripts| ResScripts

    IntroNB -->|"HTTP REST / CloudVolume API"| CloudVol
    CAVEsetup -->|"HTTP REST"| CAVE
    MeshAccess -->|"CloudVolume Mesh API"| CloudVol
    SynapseQuery -->|"CAVE Annotation API"| CAVE
    VisualSyn -->|"VTK rendering"| Trimesh

    FuncLib -->|"calls"| CloudVol
    MitoLib -->|"calls"| CloudVol
    MotifsUtils -->|"processes skeletons"| CAVE
    ResLib -->|"models & plots"| SciPy

    FuncScripts -->|"compute metrics"| FuncLib
    MitoScripts -->|"assemble data"| MitoLib
    ResScripts -->|"derive data"| ResLib

    MeshParty -->|dependency| FuncLib
    MeshParty -->|dependency| MitoLib
    Trimesh -->|dependency| MitoLib

    Neuroglancer -->|visualizes| MeshAccess
    Dash -->|interactive plots| JupyterUI

    CloudVol -->|loads volumes| Storage
    CAVE -->|stores annotations| Storage

    BinderService -->|builds| Docker

    %% Click Events
    click EnvEnvYml "https://github.com/alleninstitute/micronsbinder/blob/master/environment.yml"
    click EnvApt "https://github.com/alleninstitute/micronsbinder/blob/master/apt.txt"
    click EnvPost "https://github.com/alleninstitute/micronsbinder/tree/master/postBuild"
    click Makefile "https://github.com/alleninstitute/micronsbinder/tree/master/Makefile"
    click EnvGitignore "https://github.com/alleninstitute/micronsbinder/blob/master/.gitignore"
    click EnvTravis "https://github.com/alleninstitute/micronsbinder/blob/master/.travis.yml"
    click EnvContrib "https://github.com/alleninstitute/micronsbinder/blob/master/CONTRIBUTING.md"
    click EnvLicense "https://github.com/alleninstitute/micronsbinder/tree/master/LICENSE"
    click EnvReadme "https://github.com/alleninstitute/micronsbinder/blob/master/README.md"
    click IntroNB "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/intro/"
    click MM3Dir "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/mm3_intro/"
    click CAVEsetup "https://github.com/alleninstitute/micronsbinder/blob/master/notebooks/mm3_intro/CAVEsetup.ipynb"
    click MeshAccess "https://github.com/alleninstitute/micronsbinder/blob/master/notebooks/mm3_intro/MeshAccess.ipynb"
    click SynapseQuery "https://github.com/alleninstitute/micronsbinder/blob/master/notebooks/mm3_intro/SynapseAndAnnotationQuery.ipynb"
    click VisualSyn "https://github.com/alleninstitute/micronsbinder/blob/master/notebooks/mm3_intro/VisualizingSynapsesWithMeshes.ipynb"
    click FuncLib "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/function/lib/"
    click FuncScripts "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/function/scripts/"
    click MitoLib "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/mitochondria/lib/"
    click MitoScripts "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/mitochondria/scripts/"
    click MotifsUtils "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/motifs/utils/"
    click ResLib "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/resource_statistics/lib/"
    click ResScripts "https://github.com/alleninstitute/micronsbinder/tree/master/notebooks/vignette_analysis/resource_statistics/scripts/"

    %% Styles
    classDef env fill:#e0f7fa,stroke:#006064
    classDef config fill:#fff3e0,stroke:#e65100
    classDef envfile fill:#e1f5fe,stroke:#0277bd
    classDef orchestration fill:#f1f8e9,stroke:#33691e
    classDef ui fill:#e8f5e9,stroke:#2e7d32
    classDef notebooks fill:#f3e5f5,stroke:#6a1b9a
    classDef notebook fill:#fce4ec,stroke:#880e4f
    classDef code fill:#fffde7,stroke:#f9a825
    classDef dataaccess fill:#fff8e1,stroke:#ff6f00
    classDef analysis fill:#e8eaf6,stroke:#283593
    classDef viz fill:#f3e5f5,stroke:#4a148c
    classDef external fill:#ffebee,stroke:#b71c1c
```
