How xcompact3d/incompact3d works
Project Type and Purpose
• Xcompact3d is a high-performance Computational Fluid Dynamics (CFD) code dedicated to simulating turbulent flows with high-order accuracy.
• It supports DNS/LES and uses techniques such as the Immersed Boundary Method (IBM) and spectral-like FFT-based solvers. It is designed to run on massively parallel HPC systems (using MPI and libraries like 2DECOMP&FFT).
File Tree for Architectural Clues
• Top-level directories like “src”, “examples”, “docs”, and “tests” indicate a separation between core source code, demonstration cases, documentation, and test cases.
• The presence of CMake build files, assorted cmake directories (with special modules such as FindDECOMP2D, MPI settings, and compiler-specific flags) suggests that the code is modular and is built to support multiple compiler environments on HPC systems.
• Numerous “input” files (for various CFD cases) and configuration files (such as adios2_config.xml) imply a design that separates simulation configuration from the solver core.
• The “docs” directory contains pages related to user guide, methodology, and benchmark cases, which reflect strong support for maintainability and user interaction with the simulation framework.
• The “tests” directory shows that there is a clear framework for validation, running short simulations across different cases to ensure architecture correctness.
a. Main Components:
• Simulation Core (Solver Engines): Located under “src”, includes modules for different simulation cases (e.g., Case-ABL, Case-Channel, Case-Mixing-layer, etc.) and common solver routines (e.g., navier.f90, poisson.f90, time_integrators.f90).
• Numerical Methods and Finite Difference Schemes: Modules that implement high-order schemes, compact finite-difference methods, and treatments for dynamics (e.g., derive.f90, stretching.f90, schemes.f90).
• Immersed Boundary Methods (IBM): Specific routines (acl_*.f90, ibm.f90) to handle complex geometries and no-slip boundary conditions.
• Domain Decomposition and Parallel FFT Module: Integration with external libraries such as 2DECOMP&FFT and the cmake/FindDECOMP2D scripts. This allows for distribution of the domain and spectral solution (Poisson equation) through MPI.
• I/O and Configuration: Component that reads simulation parameters and input files (common file types in examples and tests such as .i3d). Configuration setups include ADIOS2 XML and various input files.
• Build and Integration Tools: The use of cmake setups, build scripts, and continuous integration files (.travis.yml, GitHub workflows) indicate that the project is designed with robust build and testing pipelines.
b. Relationships and Interactions:
• Data Flow: Start with simulation configuration (input files, parameters, and geometry), which is read by the I/O module and passed into the simulation core.
• Solver Core: Executes the finite-difference schemes, including the treatment for IBM and the solution of the Poisson equation via FFT, while interfacing with MPI for parallel decomposition.
• External Libraries: The simulation core leverages external modules (2DECOMP&FFT for FFT operations; ADIOS2 might be used for I/O) and interacts with compiler-specific modules.
• Testing and Examples: Separate flows where pre-set case configurations (from examples/tests directories) are submitted to the simulation engine to validate or benchmark its performance.
c. Core Architectural Patterns:
• Modularity: Clear separation between numerical solver implementations, boundary handling (IBM), I/O, and domain decomposition supports modular development and maintainability.
• Layered Structure: The design follows a layered approach – with a configuration layer (input files), a computation layer (simulation core, numerical methods, FFT, IBM), and a visualization/analysis layer (post-processing and user guides).
• Scalability and Parallelism: The integration with domain decomposition (via 2DECOMP&FFT) and MPI highlights that scalability and distributed computations are core to the design.
d. Relevant Technologies and Frameworks:
• Fortran: Primary programming language.
• MPI: Used for parallelism.
• 2DECOMP&FFT: For distributed FFTs in solving the Poisson equation.
• CMake: For building the project with various compiler configurations.
• ADIOS2 (as suggested by configuration files): For high-performance I/O.