How pnnl/neuromancer works
Project Type and Purpose
• NeuroMANCER is an open-source Python library for differentiable programming, physics-informed system identification, parametric optimization, and model-based control.
• It is not a client-server or web app but a modular software toolkit built on PyTorch (and optionally PyTorch Lightning, TorchSDE, cvxpylayers, etc.).
High-Level File-Structure Analysis
• src/neuromancer: the core Python package, subdivided into subpackages:
• analysis, arg, callbacks, constraint, dataset, dynamics, gradients, loggers, loss, modules, psl, slim, system, trainer, utils, plot, problem
• assistant/: helper scripts for generating RAG-friendly embeddings of the code/docs for an LLM assistant.
• docs/: Sphinx documentation source.
• examples/: Jupyter notebooks and scripts demonstrating use cases (ODEs, PDEs, control, KANs, etc.).
• tests/: unit tests validating core functionality.
• Configuration files (pyproject.toml, environment.yml), CI workflows, and markdown guides round out the repo.
Key Insights from README
• Core abstractions: Variable, Node, System, Problem, Loss, Trainer
• Integration points: PyTorch for tensor ops, PyTorch Lightning for training loop, optional TorchSDE for SDEs, cvxpylayers for convex layers
• Example workflows: define symbolic variables → wrap modules in Nodes → assemble into a System/Problem → hand off to Trainer → optimize parameters
Components
• PyTorch
• PyTorch Lightning
• TorchSDE (optional)
• cvxpylayers (optional)
1. Constraint & Variable Engine (constraint.py)
2. Symbolic System Builder (system.py, Node, System classes)
3. Neural Modules (modules/blocks.py, modules/activations.py, modules/rnn.py, slim/, etc.)
4. Dynamics Library (dynamics/ode.py, dynamics/sde.py, dynamics/physics.py, etc.)
5. Loss & Objective Manager (loss.py, problem.py)
6. Trainer (trainer.py, Lightning‐based training logic)
7. Callbacks & Logging (callbacks.py, loggers.py)
8. Analysis Tools (analysis/pwa_maps.py, etc.)
c. Data Layer (dataset.py and user data): feeding training/validation batches
d. User Code / Examples: imports the Core API, defines Variables, Nodes, Systems, Problems, calls Trainer
e. Documentation & Assistant: ingest scripts under assistant/, Sphinx docs under docs/ (not in runtime path but part of the ecosystem)
Interactions and Data-Flow
• User → defines Variables & Problem using Modules →
• System Builder composes Nodes into a computation graph →
• Trainer pulls batches from Dataset, runs forward on System →
• Loss Manager evaluates objectives & constraints →
• Backward/Optimizer (PyTorch) updates module parameters →
• Callbacks/Loggers record metrics →
• Analysis Tools can be invoked at any stage for post-hoc evaluation (e.g., PWA map analysis)
Architectural Patterns & Principles
• Modularity: each subpackage encapsulates a concern (e.g., modules vs. dynamics vs. loss).
• Symbolic Programming: Variables and Nodes allow for late binding of computation graphs.
• Plugin‐style Extensibility: external convex solvers, TorchSDE, custom modules can be slotted in.
• Separation of Concerns: modeling (System), optimization formulation (Problem/Loss), and execution (Trainer) are cleanly separated.