Developer Notes

Library Design

LibCEED has a single user facing API for creating and using the libCEED objects (CeedVector, CeedBasis, etc). Different Ceed backends are selected by instantiating a different Ceed object to create the other libCEED objects, in a bridge pattern. At runtime, the user can select the different backend implementations to target different hardware, such as CPUs or GPUs.

When designing new features, developers should place the function definitions for the user facing API in the header /include/ceed/ceed.h. The basic implementation of these functions should typically be placed in /interface/*.c files. The interface should pass any computationally expensive or hardware specific operations to a backend implementation. A new method for the associated libCEED object can be added in /include/ceed-impl.h, with a corresponding CEED_FTABLE_ENTRY in /interface/ceed.c to allow backends to set their own implementations of this method. Then in the creation of the backend specific implementation of the object, typically found in /backends/[impl]/ceed-[impl]-[object].c, the developer creates the backend implementation of the specific method and calls CeedSetBackendFunction() to set this implementation of the method for the backend. Any supplemental functions intended to be used in the interface or by the backends may be added to the backend API in the header /include/ceed/backend.h. The basic implementation of these functions should also be placed in /interface/*.c files.

LibCEED generally follows a “CPU first” implementation strategy when adding new functionality to the user facing API. If there are no performance specific considerations, it is generally recommended to include a basic CPU default implementation in /interface/*.c. Any new functions must be well documented and tested. Once the user facing API and the default implementation are in place and verified correct via tests, then the developer can focus on hardware specific implementations (AVX, CUDA, HIP, etc.) as necessary.

Backend Inheritance

A Ceed backend is not required to implement all libCEED objects or CeedOperator methods. There are three mechanisms by which a Ceed backend can inherit implementations from another Ceed backend.

  1. Delegation - Developers may use CeedSetDelegate() to set a general delegate Ceed object. This delegate Ceed will provide the implementation of any libCEED objects that parent backend does not implement. For example, the /cpu/self/xsmm/serial backend implements the CeedTensorContract object itself but delegates all other functionality to the /cpu/self/opt/serial backend.

  2. Object delegation - Developers may use CeedSetObjectDelegate() to set a delegate Ceed object for a specific libCEED object. This delegate Ceed will only provide the implementation of that specific libCEED object for the parent backend. Object delegation has higher precedence than delegation.

  3. Operator fallback - Developers may use CeedSetOperatorFallbackCeed() to set a Ceed object to provide any unimplemented CeedOperator methods that support preconditioning, such as CeedOperatorLinearAssemble(). The parent backend must implement the basic CeedOperator functionality. Like the delegates above, this fallback Ceed object should be created and set in the backend CeedInit function. In order to use operator fallback, the parent backend and fallback backend must use compatible E-vector and Q-vector layouts. For example, /gpu/cuda/gen falls back to /gpu/cuda/ref for missing CeedOperator preconditioning support methods. If an unimplemented method is called, then the parent /gpu/cuda/gen Ceed object uses its fallback /gpu/cuda/ref Ceed object to create a clone of the CeedOperator. This clone CeedOperator is then used for the unimplemented preconditioning support methods.

Backend Families

There are 4 general ‘families’ of backend implementations. As internal data layouts are specific to backend families, it is generally not possible to delegate between backend families.

CPU Backends

The basic CPU with the simplest implementation is /cpu/self/ref/serial. This backend contains the basic implementations of most objects that other backends rely upon. Most of the other CPU backends only update the CeedOperator and CeedTensorContract objects.

The /cpu/self/ref/blockend and /cpu/self/opt/* backends delegate to the /cpu/self/ref/serial backend. The /cpu/self/ref/blocked backend updates the CeedOperator to use an E-vector and Q-vector ordering when data for 8 elements are interlaced to provide better vectorization. The /cpu/self/opt/* backends update the CeedOperator to apply the action of the operator in 1 or 8 element batches, depending upon if the blocking strategy is used. This reduced the memory required to utilize this backend significantly.

The /cpu/self/avx/* and /cpu/self/xsmm/* backends delegate to the corresponding /cpu/self/opt/* backends. These backends update the CeedTensorContract objects using AVX intrinsics and libXSMM functions, respectively.

The /cpu/self/memcheck/* backends delegate to the /cpu/self/ref/* backends. These backends replace many of the implementations with methods that include more verification checks and a memory management model that more closely matches the memory management for GPU backends. These backends rely upon the Valgrind Memcheck tool and Valgrind headers.

GPU Backends

The CUDA, HIP, and SYCL backend families all follow similar designs. The CUDA and HIP backends are very similar, with minor differences. While the SYCL backend was based upon the CUDA and HIP backends, there are more internal differences to accommodate OpenCL and Intel hardware.

The /gpu/*/ref backends provide basic functionality. In these backends, the operator is applied in multiple separate kernel launches, following the libCEED operator decomposition, where first CeedElemRestriction kernels map from the L-vectors to E-vectors, then CeedBasis kernels map from the E-vectors to Q-vectors, then the CeedQFunction kernel provides the action of the user quadrature point function, and the transpose CeedBasis and CeedElemRestriction kernels are applied to go back to the E-vectors and finally the L-vectors. These kernels apply to all points across all elements in order to maximize the amount of work each kernel launch has. Some of these kernels are compiled at runtime via NVRTC, HIPRTC, or OpenCL RTC.

The /gpu/*/shared backends delegate to the corresponding /gpu/*/ref backends. These backends use shared memory to improve performance for the CeedBasis kernels. All other libCEED objects are delegated to /gpu/*/ref. These kernels are compiled at runtime via NVRTC, HIPRTC, or OpenCL RTC.

The /gpu/*/gen backends delegate to the corresponding /gpu/*/shared backends. These backends write a single comprehensive kernel to apply the action of the CeedOperator, significantly improving performance by eliminating intermediate data structures and reducing the total number of kernel launches required. This kernel is compiled at runtime via NVRTC, HIPRTC, or OpenCL RTC.

The /gpu/*/magma backends delegate to the corresponding /gpu/cuda/ref and /gpu/hip/ref backends. These backends provide better performance for CeedBasis kernels but do not have the improvements from the /gpu/*/gen backends for CeedOperator.

Internal Layouts

Ceed backends are free to use any E-vector and Q-vector data layout (including never fully forming these vectors) so long as the backend passes the t5** series tests and all examples. There are several common layouts for L-vectors, E-vectors, and Q-vectors, detailed below:

  • L-vector layouts

    • L-vectors described by a standard CeedElemRestriction have a layout described by the offsets array and comp_stride parameter. Data for node i, component j, element k can be found in the L-vector at index offsets[i + k*elem_size] + j*comp_stride.

    • L-vectors described by a strided CeedElemRestriction have a layout described by the strides array. Data for node i, component j, element k can be found in the L-vector at index i*strides[0] + j*strides[1] + k*strides[2].

  • E-vector layouts

    • If possible, backends should use CeedElemRestrictionSetELayout() to use the t2** tests. If the backend uses a strided E-vector layout, then the data for node i, component j, element k in the E-vector is given by i*layout[0] + j*layout[1] + k*layout[2].

    • Backends may choose to use a non-strided E-vector layout; however, the t2** tests will not function correctly in this case and these tests will need to be marked as allowable failures for this backend in the test suite.

  • Q-vector layouts

    • When the size of a CeedQFunction field is greater than 1, data for quadrature point i component j can be found in the Q-vector at index i + Q*j, where Q is the total number of quadrature points in the Q-vector. Backends are free to provide the quadrature points in any order.

    • When the CeedQFunction field has emode CEED_EVAL_GRAD, data for quadrature point i, component j, derivative k can be found in the Q-vector at index i + Q*j + Q*num_comp*k.

    • Backend developers must take special care to ensure that the data in the Q-vectors for a field with emode CEED_EVAL_NONE is properly ordered when the backend uses different layouts for E-vectors and Q-vectors.

CeedVector Array Access

Backend implementations are expected to separately track ‘owned’ and ‘borrowed’ memory locations. Backends are responsible for freeing ‘owned’ memory; ‘borrowed’ memory is set by the user and backends only have read/write access to ‘borrowed’ memory. For any given precision and memory type, a backend should only have ‘owned’ or ‘borrowed’ memory, not both.

Backends are responsible for tracking which memory locations contain valid data. If the user calls CeedVectorTakeArray() on the only memory location that contains valid data, then the CeedVector is left in an invalid state. To repair an invalid state, the user must set valid data by calling CeedVectorSetValue(), CeedVectorSetArray(), or CeedVectorGetArrayWrite().

Some checks for consistency and data validity with CeedVector array access are performed at the interface level. All backends may assume that array access will conform to these guidelines:

Shape

Backends often manipulate tensors of dimension greater than 2. It is awkward to pass fully-specified multi-dimensional arrays using C99 and certain operations will flatten/reshape the tensors for computational convenience. We frequently use comments to document shapes using a lexicographic ordering. For example, the comment

// u has shape [dim, num_comp, Q, num_elem]

means that it can be traversed as

for (d = 0; d < dim; d++) {
  for (c = 0; c < num_comp; c++) {
    for (q = 0; q < Q; q++) {
      for (e = 0; e < num_elem; e++) {
        u[((d*num_comp + c)*Q + q)*num_elem + e] = ...

This ordering is sometimes referred to as row-major or C-style. Note that flattening such as

// u has shape [dim, num_comp, Q*num_elem]

and

// u has shape [dim*num_comp, Q, num_elem]

are purely implicit – one just indexes the same array using the appropriate convention.

restrict Semantics

QFunction arguments can be assumed to have restrict semantics. That is, each input and output array must reside in distinct memory without overlap.

Style Guide

Please check your code for style issues by running

make format

In addition to those automatically enforced style rules, libCEED tends to follow the following code style conventions:

  • Variable names: snake_case

  • Strut members: snake_case

  • Function and method names: PascalCase or language specific style

  • Type names: PascalCase or language specific style

  • Constant names: CAPS_SNAKE_CASE or language specific style

In general, variable and function names should avoid abbreviations and err on the side of verbosity to improve readability.

Also, documentation files should have one sentence per line to help make git diffs clearer and less disruptive.

Function Conventions

Naming

All functions in the libCEED library should be prefixed by Ceed and generally take a Ceed object as its first argument. If a function takes, for example, a CeedOperator as its first argument, then it should be prefixed with CeedOperator.

Style

Functions should adhere mostly to the PETSc function style, specifically:

  1. All local variables of a particular type (for example, CeedInt) should be listed on the same line if possible; otherwise, they should be listed on adjacent lines. For example,

// Correct
CeedInt   a, b, c;
CeedInt  *d, *e;
CeedInt **f;

// Incorrect
CeedInt a, b, c, *d, *e, **f;

  1. Local variables should be initialized in their declaration when possible.

  2. Nearly all functions should have a return type of int and return a CeedErrorType to allow for error checking.

  3. All functions must start with a single blank line after the local variable declarations.

  4. All libCEED function calls must have their return value checked for errors using the CeedCall() or CeedCallBackend() macro. This should be wrapped around the function in question.

  5. In libCEED functions, variables must be declared at the beginning of the code block (C90 style), never mixed in with code. However, when variables are only used in a limited scope, it is encouraged to declare them in that scope.

  6. Do not put a blank line immediately before return CEED_ERROR_SUCCESS;.

  7. All libCEED functions must use Doxygen comment blocks before their definition (not declaration). The block should begin with /** and end with **/, each on their own line. The block should be indented by two spaces and should contain an @brief tag and description, a newline, a line stating whether the function is collective, a newline, @param tags for each parameter, a newline, and a @return line formatted exactly as in the example below. All parameter lines in the Doxygen block should be formatted such that parameter names and descriptions are aligned. There should be a exactly one space between @param[dir] (where dir is in, out, or in,out) and the parameter name for the closest pair, as well as between the parameter name and description. For example:

/**
  @brief Initialize a `Ceed` context to use the specified resource.

  Note: Prefixing the resource with "help:" (e.g. "help:/cpu/self") will result in @ref CeedInt() printing the current libCEED version number and a list of current available backend resources to `stderr`.

  @param[in]  resource Resource to use, e.g., "/cpu/self"
  @param[out] ceed     The library context

  @return An error code: 0 - success, otherwise - failure

  @ref User

  @sa CeedRegister() CeedDestroy()
**/
int CeedInit(const char *resource, Ceed *ceed) {

  1. Function declarations should include parameter names, which must exactly match those in the function definition.

  2. External functions, i.e. those used in tests or examples, must have their declarations prefixed with CEED_EXTERN. All other functions should have their declarations prefixed with CEED_INTERN. Function definitions should have neither.

Clang-tidy

Please check your code for common issues by running

make tidy

which uses the clang-tidy utility included in recent releases of Clang. This tool is much slower than actual compilation (make -j8 parallelism helps). To run on a single file, use

make interface/ceed.c.tidy

for example. All issues reported by make tidy should be fixed.

Include-What-You-Use

Header inclusion for source files should follow the principal of ‘include what you use’ rather than relying upon transitive #include to define all symbols.

Every symbol that is used in the source file foo.c should be defined in foo.c, foo.h, or in a header file #included in one of these two locations. Please check your code by running the tool include-what-you-use to see recommendations for changes to your source. Most issues reported by include-what-you-use should be fixed; however this rule is flexible to account for differences in header file organization in external libraries. If you have include-what-you-use installed in a sibling directory to libCEED or set the environment variable IWYU_CC, then you can use the makefile target make iwyu.

Header files should be listed in alphabetical order, with installed headers preceding local headers and ceed headers being listed first. The ceed-f64.h and ceed-f32.h headers should only be included in ceed.h.

#include <ceed.h>
#include <ceed/backend.h>
#include <stdbool.h>
#include <string.h>
#include "ceed-avx.h"