# alleninstitute/dipde architecture

> Architecture diagram of the GitHub repository alleninstitute/dipde, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alleninstitute/dipde
- Repository: https://github.com/alleninstitute/dipde

## Overview

##### Project Type and Purpose
- Type: A Python scientific library/package for simulating mesoscale neuronal population dynamics.
- Purpose: Provide a high-performance engine (core internals) plus multiple interfaces to run distributed/networked simulations, analyze results, and generate visualizations.

##### File Structure
- Top-Level Packaging & Tooling
– setup.py, setup.cfg: Python packaging metadata
– environment.yml, requirements.txt: Conda/Pip dependencies
– Dockerfile, docker/…: Containerized runtime environments (Python2.7, Python3.6)
– .travis.yml, Makefile, conda/, conda_test.sh: CI/build scripts
- Core Library (dipde/)
– internals/: Core simulation engine (network, populations, connections, solvers)
– interfaces/: Adapters to external frameworks (MPI, ZMQ, NetworkX, NEuromL, NWB, pandas)
– experimental/: Prototypes & distributed solvers
– profiling/: Instrumentation hooks
– visualization/: Plotting and interactive sandbox
– examples/: User‐level demo scripts showing typical usage
– test/: Unit and integration tests
- Documentation (doc/, doc_template/)
– Sphinx sources and generated HTML

##### README for Insights
- Confirms it’s a simulation platform solving population density equations.
- Encourages community contributions.

##### Main Components & Their Relationships
a. Core Simulation Engine (dipde/internals)
– SimulationConfiguration → Simulation → Network → Populations → ConnectionDistribution → Solver
b. Interface Layer (dipde/interfaces)
– Wraps core engine for parallel (MPI, ZMQ), graph input (NetworkX), data I/O (pandas, NWB), model import/export (NeuroML)
c. Experimental Extensions (dipde/experimental)
– Builds on core+interfaces for distributed backends, fixed‐point analyses
d. User/API Layer (dipde/examples, dipde/visualization)
– Example scripts call core & interfaces to set up models and launch runs
– Visualization sandbox consumes simulation outputs
e. Deployment & Packaging
– Conda recipes, Docker containers
– CI pipelines (Travis), Makefiles, test harness

##### Architectural Patterns & Design Principles
- Modular layering: Core engine separated from interfaces, examples, and experimental code.
- Adapter pattern: Each interface subpackage adapts the core to a specific external system.
- Pluggable backend: Experimental/distributed backends can be swapped in via configuration.
- Test‐driven: test/ directory validates each layer.
- Documentation pipeline: Sphinx docs auto‐build from source.

## Diagram (Mermaid)

```mermaid
graph TB
    %% User/API Layer
    subgraph "User/API Layer" 
        direction TB
        EXP_SOLVER["ExponentialSolver.py"]:::user
        CORTICAL["cortical_column.py"]:::user
        EXC_INH["excitatory_inhibitory.py"]:::user
        FEEDFORWARD["feedforward_exponential_distribution.py"]:::user
        SINGLEPOP["singlepop*.py"]:::user
        VIS_SANDBOX["Visualization Sandbox"]:::user
    end

    %% Interfaces Layer
    subgraph "Interfaces Layer"
        direction TB
        AM["access_matrices.py"]:::interface
        subgraph "MPI Adapter"
            direction TB
            MPI_BENCH["benchmark.py"]:::interface
            MPI_RUN["run_benchmark.py"]:::interface
            MPI_SYNC["synchronizationharness.py"]:::interface
        end
        ZMQ["zmq/"]:::interface
        NX["networkx/"]:::interface
        subgraph "NeuroML Adapter"
            direction TB
            NML_FILE["dipde_2.net.nml"]:::interface
            NML_SANDBOX["sandbox.py"]:::interface
        end
        NWB["nwb/"]:::interface
        PANDAS["pandas/"]:::interface
    end

    %% Core Engine
    subgraph "Core Simulation Engine" 
        direction TB
        SC["SimulationConfiguration"]:::core
        SIM["Simulation"]:::core
        NET["Network"]:::core
        CON["Connection"]:::core
        COND["ConnectionDistribution"]:::core
        CONDC["ConnectionDistributionCollection"]:::core
        IP["InternalPopulation"]:::core
        EP["ExternalPopulation"]:::core
        FR["FiringRateOrganizer"]:::core
        UTIL["Utilities"]:::core
    end

    %% Experimental Extensions
    subgraph "Experimental Extensions"
        direction TB
        subgraph "Distributed"
            direction TB
            DIST_CFG["distributedconfiguration.py"]:::exp
            DIST_SYNC_EXP["synchronizationharness.py"]:::exp
            DIST_MONGO["distributed_mongo.py"]:::exp
            DIST_SANDBOX["distributed_sandbox.py"]:::exp
            DIST_ZMQ["distributed_zmq.py"]:::exp
        end
        subgraph "Fixed Point Analysis"
            direction TB
            NINT["nullspace_intuition.py"]:::exp
            PARAM_JAC["parameter_jacobian_ei.py"]:::exp
            RE_EX["recurrent_excitatory.py"]:::exp
        end
        MONGOEXT["mongodistributedconfiguration.py"]:::exp
        PYNEST["pynest/"]:::exp
        XTEST["xtest_distributed.py"]:::exp
        LAUNCH["launch.sh"]:::exp
    end

    %% Deployment & Packaging
    subgraph "Deployment & Packaging"
        direction TB
        SETUP_PY["setup.py"]:::deploy
        ENV_YML["environment.yml"]:::deploy
        DOCKERFILE["Dockerfile"]:::deploy
        TRAVIS[".travis.yml"]:::deploy
        MAKEFILE["Makefile"]:::deploy
        CONDA["conda/"]:::deploy
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        DOC_SRC["doc/source/"]:::doc
        DOC_BUILD["doc/build/html/"]:::doc
        DOC_TPL["doc_template/"]:::doc
    end

    %% Relationships
    EXP_SOLVER -->|uses| AM
    CORTICAL -->|uses| MPI_RUN
    EXC_INH -->|uses| ZMQ
    FEEDFORWARD -->|uses| NX
    SINGLEPOP -->|uses| NML_SANDBOX
    VIS_SANDBOX -->|consumes results| PANDAS

    AM -->|adapts| SC
    MPI_RUN -->|adapts| SIM
    MPI_SYNC -->|sync| SIM
    ZMQ -->|adapts| SIM
    NX -->|imports| NET
    NML_SANDBOX -->|imports| NET
    PANDAS -->|I/O| UTIL

    SC -->|"configures"| SIM
    SIM -->|"builds"| NET
    NET -->|"connects via"| CON
    CON -->|"distribution"| COND
    COND -->|"collection"| CONDC
    CONDC -->|"populations"| IP
    IP -->|"outputs"| FR
    FR -->|"helpers"| UTIL

    SIM -->|"exports data"| PANDAS
    SIM -->|"exports file"| NWB

    EXP -->CORE
    EXP -->INTERFACES

    subgraph Legend
        direction LR
        LC[Rectangles: Modules]:::legend
        LS[Rounded: Scripts]:::legend
        LD[Cylinders: Data Stores]:::legend
        LCLO[Clouds: External Systems]:::legend
    end

    %% Click Events
    click SC "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/simulationconfiguration.py"
    click SIM "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/simulation.py"
    click NET "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/network.py"
    click CON "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/connection.py"
    click COND "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/connectiondistribution.py"
    click CONDC "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/connectiondistributioncollection.py"
    click IP "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/internalpopulation.py"
    click EP "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/externalpopulation.py"
    click FR "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/firingrateorganizer.py"
    click UTIL "https://github.com/alleninstitute/dipde/blob/dev/dipde/internals/utilities.py"

    click AM "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/access_matrices.py"
    click MPI_BENCH "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/mpi/benchmark.py"
    click MPI_RUN "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/mpi/run_benchmark.py"
    click MPI_SYNC "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/mpi/synchronizationharness.py"
    click NML_FILE "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/neuroml/dipde_2.net.nml"
    click NML_SANDBOX "https://github.com/alleninstitute/dipde/blob/dev/dipde/interfaces/neuroml/sandbox.py"

    click DIST_CFG "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/distributed/distributedconfiguration.py"
    click DIST_SYNC_EXP "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/distributed/synchronizationharness.py"
    click DIST_MONGO "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/distributed_mongo.py"
    click DIST_SANDBOX "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/distributed_sandbox.py"
    click DIST_ZMQ "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/distributed_zmq.py"
    click NINT "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/fixed_point_analysis/nullspace_intuition.py"
    click PARAM_JAC "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/fixed_point_analysis/parameter_jacobian_ei.py"
    click RE_EX "https://github.com/alleninstitute/dipde/blob/dev/dipde/experimental/fixed_point_analysis/recurrent_excitatory.py"

    click EXP_SOLVER "https://github.com/alleninstitute/dipde/blob/dev/dipde/examples/ExponentialSolver.py"
    click CORTICAL "https://github.com/alleninstitute/dipde/blob/dev/dipde/examples/cortical_column.py"
    click EXC_INH "https://github.com/alleninstitute/dipde/blob/dev/dipde/examples/excitatory_inhibitory.py"
    click FEEDFORWARD "https://github.com/alleninstitute/dipde/blob/dev/dipde/examples/feedforward_exponential_distribution.py"
    click SINGLEPOP "https://github.com/alleninstitute/dipde/blob/dev/dipde/examples/singlepop*.py"
    click VIS_SANDBOX "https://github.com/alleninstitute/dipde/blob/dev/dipde/visualization/sandbox.py"

    click SETUP_PY "https://github.com/alleninstitute/dipde/blob/dev/setup.py"
    click ENV_YML "https://github.com/alleninstitute/dipde/blob/dev/environment.yml"
    click DOCKERFILE "https://github.com/alleninstitute/dipde/tree/dev/Dockerfile"
    click TRAVIS "https://github.com/alleninstitute/dipde/blob/dev/.travis.yml"
    click MAKEFILE "https://github.com/alleninstitute/dipde/tree/dev/Makefile"
    click CONDA "https://github.com/alleninstitute/dipde/tree/dev/conda/"

    click DOC_SRC "https://github.com/alleninstitute/dipde/tree/dev/doc/source/"
    click DOC_BUILD "https://github.com/alleninstitute/dipde/tree/dev/doc/build/html/"
    click DOC_TPL "https://github.com/alleninstitute/dipde/tree/dev/doc_template/"

    %% Styles
    classDef core fill:#f9f,stroke:#333,stroke-width:1px;
    classDef interface fill:#9f9,stroke:#333,stroke-width:1px;
    classDef exp fill:#ff9,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef user fill:#9ff,stroke:#333,stroke-width:1px,stroke-radius: 5px;
    classDef deploy fill:#ccc,stroke:#333,stroke-width:1px;
    classDef doc fill:#cfc,stroke:#333,stroke-width:1px;
    classDef legend fill:#fff,stroke:#000,stroke-width:1px,stroke-dasharray: 2 2;
```
