How computationalflowphysics/iblgf-amr works
1. Repository purpose and top-level shape
This is a C++17 scientific computing library/application for immersed-boundary incompressible flow on unbounded domains, built around lattice Green’s functions (LGF). The code is organized as a reusable library under include/iblgf, with a small amount of compiled implementation in src, plus many scenario-specific executables/tests under tests and benchmarking drivers under bench.
2. Public library boundary vs. implementation
The include/iblgf tree is the main public API surface. It contains the domain model, solvers, operators, I/O, interpolation, linear algebra wrappers, and simulation/setup abstractions. The src tree provides non-header implementations for shared kernels such as LGF lookup tables, convolution, GPU convolution, and math helpers.
3. Core simulation abstractions
The simulation flow is anchored by simulation.hpp, the setup_* headers, and problem-specific test drivers. A setup object configures geometry, physics, boundary conditions, and algorithm choice, then feeds the solver stack. This is the boundary where application-specific cases enter the library.
4. Domain representation and adaptive mesh infrastructure
include/iblgf/domain holds the main spatial/data model:
- octree for adaptive hierarchical mesh indexing and traversal,
- dataFields for blocks, nodes, views, and field storage,
- decomposition and mpi for distributed ownership and task exchange,
- ib / ib_communicator for immersed-boundary coupling.
This layer is the backbone for how state is partitioned, refined, communicated, and accessed across ranks.
5. Parallel execution model
MPI is the primary scaling boundary, with explicit client/server/task abstractions in domain/mpi and domain/decomposition. OpenMP variants appear in several solver headers, and CUDA is used for selected convolution/Poisson paths. The architecture therefore supports multiple parallel backends, but MPI distributed memory is the main coordination layer.
6. LGF and elliptic operator machinery
The lgf, operators, interpolation, and utilities/convolution* modules implement the numerical heart of the code: applying lattice Green’s function operators, interpolation between grids, and convolution-based solves. These pieces provide the fast unbounded-domain elliptic solve path used by Poisson and Helmholtz-related workflows.
7. Solver stack and numerical workflows
include/iblgf/solver contains several solver families:
- Poisson and Helmholtz solvers, including OpenMP variants,
- time-integration and implicit methods (ifherk, Helmholtz-FFT, linear and Helmholtz-coupled variants),
- Newton and stability solvers,
- modal analysis and matrix-free components,
- direct solver hooks such as MKL PARDISO and ScaLAPACK interfaces.
This layer consumes the domain/operator abstractions and produces the actual physical updates, eigen/spectral analyses, or linear-system solutions.
8. Linear algebra and external numerical backends
linalg, tensor, and solver-specific wrappers abstract numerical arrays and BLAS-like operations, mostly via xtensor/xtensor-blas. The repository also integrates with heavy external backends when needed: FFTW, MKL, ScaLAPACK, PETSc, and SLEPc. These are not just optional utilities; they define alternate solver paths and are exercised by dedicated tests.
9. I/O, restart, and reproducibility
The IO subsystem, especially IO/chombo, handles HDF5-based structured output and restart data. The README describes timestamped run directories, restart files, and resume support, indicating that long-running simulations are checkpointable and reproducible. This is an important architectural boundary between compute kernels and persistent run metadata/results.
10. Case-specific test executables as integration examples
The tests tree is effectively a catalog of end-to-end application drivers rather than unit tests alone. It includes Poisson, Helmholtz, Newton, immersed-boundary flow, stability, spectral, FFT/MPI, GPU, and PETSc/SLEPc cases. These tests show how the library is composed for different physics and backend combinations, and they are the clearest examples of the data flow from config files → setup → domain/communication → solver → HDF5 reference comparison.
11. Benchmarking and experiment generation tooling
The bench directory contains Python scripts and case generators for IF-HERK, vortex ring, Poisson GPU, and related performance studies. These scripts sit outside the core library and are used to mass-produce inputs, launch jobs, and analyze results, which makes them useful for performance/regression architecture but not part of the runtime solver path.
12. Build, packaging, and CI environment
The project is CMake-based and includes a FindFFTW.cmake, a package config template, GitHub Actions workflows for test/benchmark/containerization, and Docker helper scripts. This indicates a portable HPC-oriented build/deploy model: local CMake builds, CI validation, and containerized execution for reproducibility.
13. What an architecture diagram should emphasize
The main diagram should show: configuration/setup entering at the top, a distributed adaptive domain and data-field layer in the middle, solver/operator/LGF kernels as the compute core, MPI/OpenMP/CUDA as execution backends, and HDF5 restart/output at the bottom. Optional external solver backends (FFTW, MKL, PETSc/SLEPc, ScaLAPACK) should be shown as interchangeable integration points rather than primary subsystems.
Main entry points: poisson.cpp Drivers · domain.hpp Domain · operators.hpp Operators · simulation.hpp Simulation
Components 26
Public API
Header-only library surface exposed from include/iblgf
- Simulation · Top-level simulation orchestration and lifecycle
include/iblgf/simulation.hpp - Domain · Umbrella domain model for mesh, data, communication, and immersed boundary coupling
include/iblgf/domain/domain.hpp
Domain + Parallelism
Adaptive mesh, distributed ownership, and communication layers
- Octree · Hierarchical indexing and traversal for refined grids
include/iblgf/domain/octree/tree.hpp - Data fields · Blocks, nodes, views, and field containers
include/iblgf/domain/dataFields/datafield.hpp - Decomposition · Distributed partitioning and client/server task exchange
include/iblgf/domain/decomposition/decomposition.hpp - MPI layer · Task, halo, and registry helpers for distributed execution
include/iblgf/domain/mpi/task_manager.hpp - Immersed boundary · IBM coupling and boundary communication
include/iblgf/domain/ib_communicator.hpp
Numerical Core
LGF operators, interpolation, convolutions, and linear algebra wrappers
- LGF · Lattice Green's function and Helmholtz lookup tables
include/iblgf/lgf/lgf.hpp - Operators · Operator assembly and application interfaces
include/iblgf/operators/operators.hpp - Interpolation · Interpolation between adaptive and cell-centered representations
include/iblgf/interpolation/interpolation.hpp - Convolution · CPU and GPU convolution kernels used by LGF-based solves
include/iblgf/utilities/convolution.hpp - Linalg · Vector and tensor abstractions over xtensor-style backends
include/iblgf/linalg/linalg.hpp
Solvers
Poisson, Helmholtz, time-integration, Newton, stability, and spectral workflows
- Poisson · Poisson solver family including OpenMP variants
include/iblgf/solver/poisson/poisson.hpp - Helmholtz · Helmholtz solver family and FFT-based time-integration path
include/iblgf/solver/time_integration/HelmholtzFFT.hpp - IF-HERK · Implicit-explicit time integration and Helmholtz-coupled variants
include/iblgf/solver/time_integration/ifherk.hpp - Newton · Newton and Jacobian-based nonlinear solution workflows
include/iblgf/solver/Newton/NewtonMethod.hpp - Stability · Stability and modal analysis routines
include/iblgf/solver/Stability_solver/Stability_solver.hpp - Direct solve · MKL PARDISO and ScaLAPACK direct solver hooks
include/iblgf/solver/DirectSolver/MKLPardiso_solve.hpp
I/O
Output, checkpointing, and restart persistence
- I/O · Run output, parallel streams, and checkpoint/restart support
include/iblgf/IO/output_directory.hpp - Chombo HDF5 · Structured HDF5 output and restart serialization
include/iblgf/IO/chombo/chombo.hpp
Runtime Setup
Simulation configuration and problem-specific entry points
- Setup base · Common simulation setup interface
setups/setup_base.hpp - Problem setups · Helmholtz, Newton, linear, and stability setup specializations
setups/setup_helmholtz.hpp
Drivers
Integration-style executables and backend-specific cases
- Drivers · Representative end-to-end executables and backend exercises
tests/poisson/poisson.cpp - Benchmarks · Case generators, plotting, and performance study scripts
bench/poisson_gpu/make_jobs.py
Build + Tooling
Build system, CI, containerization, and benchmark tooling
- Build system · CMake project configuration and package discovery
CMakeLists.txt - CI + containers · GitHub Actions workflows and Docker helpers
.github/workflows/test.yml
Connections 32
- Simulation → Setup base: uses · Simulation lifecycle is configured through a setup abstraction
- Setup base → Problem setups: specialized by · Concrete problem setups provide case-specific parameters and physics
- Problem setups → Domain: configure · Setups create and initialize the domain model
- Domain → Octree: contains · Adaptive spatial indexing is a core part of the domain
- Domain → Data fields: contains · Field storage and views live under the domain model
- Domain → Decomposition: contains · Distributed ownership and task decomposition are domain concerns
- Domain → MPI layer: communicates via · MPI task and halo plumbing coordinate distributed execution
- Domain → Immersed boundary: couples · Immersed-boundary coupling bridges geometry and solver state
- Domain → Operators: drives · Domain state feeds discrete operators
- Operators → LGF: applies · Operators are built on lattice Green's function kernels
- Operators → Interpolation: uses · Operators rely on interpolation between representations
- LGF → Convolution: implemented by · LGF application depends on convolution kernels
- Convolution → Linalg: builds on · Convolution and solver kernels use linear algebra abstractions
- Poisson → Operators: consumes · Poisson solves use the operator layer
- Helmholtz → LGF: consumes · Helmholtz paths use LGF kernels and lookup tables
- IF-HERK → Helmholtz: couples to · IF-HERK variants embed Helmholtz solves inside time stepping
- Newton → Operators: linearizes · Newton workflows assemble matrix-free or explicit Jacobian operators
- Stability → Linalg: analyzes · Stability and modal analysis rely on linear algebra backends
- Direct solve → Linalg: wraps · Direct solver interfaces sit behind the linear algebra layer
- Drivers → Simulation: exercise · Integration drivers instantiate the full simulation stack
- Drivers → Poisson: covers · Dedicated cases validate Poisson workflows
- Drivers → Helmholtz: covers · Dedicated cases validate Helmholtz workflows
- Drivers → IF-HERK: covers · Dedicated cases validate time-integration workflows
- Drivers → Newton: covers · Dedicated cases validate Newton workflows
- Drivers → Stability: covers · Dedicated cases validate stability and spectral workflows
- Drivers → Direct solve: covers · Dedicated cases exercise MKL and ScaLAPACK paths
- Drivers → Chombo HDF5: compares · Cases use HDF5 reference data and restart/output checks
- I/O → Chombo HDF5: implements · Chombo HDF5 is the primary persistence backend
- Simulation → I/O: writes · Simulation results and checkpoints flow to I/O
- Benchmarks → Drivers: extends · Benchmark scripts generate or wrap case definitions
- Build system → Drivers: builds · CMake orchestrates the executable catalog
- CI + containers → Build system: validates · Workflows run builds, tests, and containerized checks