How yigongqin/cupf works
Project Type and Purpose
• This is a high-performance computing (HPC) scientific simulation toolkit (“cuPF”) for phase-field modeling of microstructure evolution in metals.
• Core components include C++/CUDA solvers (2D/3D, grain-scale and dendrite-scale), Python scripts for orchestration, input generation, post-processing, and MPI integration for multi-GPU scaling.
File Structure Analysis
• Top-level modules:
• 2DdendriteMPI (DNS & line solvers)
• 2Dgrain (various 2D grain-scale solvers and ML-grain submodules)
• 3Dgrain (single-GPU 3D solver)
• 3DgrainMPI (CUDA-aware MPI 3D solver with Python front-end)
• AdditiveManufacturing (MPI-enabled 3D melt-pool simulation)
• Each module follows the same pattern:
• C++/CUDA source (main.cpp or *.cu) + headers
• Makefile for build (variables for CXX, CUDA_PATH, HDF5, MPICXX, MPI_INC)
• Python scripts for input/parameter setups and data transformation (e.g., grains3D.py, saveVTKdata.py)
• Slurm batch scripts (.slurm) under ML_Grain and MPI modules for job submission
• Common dependencies:
• CUDA runtime and Toolkit
• MPI with CUDA-awareness (e.g., MVAPICH2-GDR)
• HDF5 library for output
• Python libraries (requirements.txt)
Key Insights from README
• Build workflow revolves around Makefiles—set environment variables, run make.
• Runtime workflows:
• Direct binary invocation for non-MPI modules (e.g., ./phase_field input_file).
• Python front-end (.py scripts) that set up parameters, invoke MPI-enabled binaries, capture HDF5 output.
• Post-processing via saveVTKdata.py → VTK files → visualization in ParaView.
Components
a. Main Components
• Development Environment: source code, Makefiles, Python scripts
• Build Toolchain: CXX/CUDA compiler, MPI compiler wrapper, HDF5 includes/libs
• Runtime Environment: HPC cluster (multi-node, multi-GPU), job scheduler (Slurm)
• Simulation Engines:
• 2D solvers (2DdendriteMPI, 2Dgrain)
• 3D solvers (3Dgrain, 3DgrainMPI, AdditiveManufacturing)
• Orchestration Layer: Python scripts for input generation, parameter sweeps, MPI invocation
• Data Layer:
• HDF5 output files
• VTK conversion (saveVTKdata.py)
• Visualization tools (ParaView)
b. Relationships and Interactions
• Developers write/edit C++/CUDA & Python code → build via Makefile → produce executables.
• Python front-end triggers the executables with command-line parameters (material properties, geometry).
• In MPI modules, Python invokes mpirun/mpiexec across GPUs.
• Executables link to CUDA, MPI, HDF5 libraries → write simulation results to HDF5.
• Post-processing Python reads HDF5 → outputs VTK → external viewer.
• Slurm scripts wrap Python/binary calls for batch submission.
c. Architectural Patterns & Principles
• Modular codebase: separate modules for 2D vs. 3D vs. additive manufacturing.
• Layered architecture:
1. Source layer (C++/CUDA + Python)
2. Build layer (Makefiles, environment variables)
3. Execution layer (HPC cluster, MPI, job scheduler)
4. Data layer (HDF5, VTK, visualization)
• Plugin-style parameterization: Python scripts encapsulate geometry and simulation options.
• Reusable core solvers extended by domain-specific wrappers (e.g., lineModel, DNS, moving-domain).
d. Key Technologies and Libraries
• CUDA Toolkit (cuBLAS, custom devicefunc.cu_inl)
• MPI (CUDA-aware variants: MVAPICH2-GDR)
• HDF5 (data serialization)
• Python (NumPy, h5py, argparse)
• Slurm (batch scheduling)
• ParaView (visualization)
Detailed Capture of Essential Architectural Elements
• Parallel execution via MPI and domain decomposition (multi-GPU).
• How Python scripts abstract simulation parameters and call underlying solvers.
• Both “2DdendriteMPI” and “AdditiveManufacturing” as representative modules.
• End-to-end workflow: from source code through visualization.