cuBLAS

Overview

  • Tutorial: 30 min

  1. Understand the cuBLAS library and its role in CUDA programming.

  2. Learn how to perform basic matrix operations using cuBLAS.

  3. Explore advanced features of cuBLAS for performance optimization.

The cuBLAS library is NVIDIA’s implementation of the Basic Linear Algebra Subprograms (BLAS) on NVIDIA GPUs. It provides highly optimized routines for performing basic linear algebra operations such as matrix multiplication, vector addition, and more. cuBLAS is designed to take advantage of the parallel processing capabilities of NVIDIA GPUs, making it a powerful tool for high-performance computing applications. cuBLAS is particularly useful for applications that require efficient matrix operations like scientific computing.

cuBLAS provides highly optimized routines for common linear algebra operations, such as:

  • Vector operations (Level 1 BLAS)

  • Matrix-vector operations (Level 2 BLAS)

  • Matrix-matrix operations (Level 3 BLAS)

Matrix Multiplication using cuBLAS

The operation is: C = alpha * A * B + beta * C, where A, B, and C are matrices.

Important

The header file for cuBLAS is cublas_v2.h, and the library is linked with -lcublas.

Column and Row Major Order

Row-Major Order stores matrix row by row in memory while Column-Major Order Stores matrix column by column in memory.

Example for matrix A[2][2] = {{1, 2}, {3, 4}} is stored as

Row-major:

1

2

3

4

and Column-major:

1

3

2

4

cuBLAS uses column-major ordering (like Fortran), while C/C++ uses row-major. Use IDX2C or manually transpose if necessary.

1#define IDX2C(i,j,ld) (((j)*(ld))+(i))  // Macro to index column-major order

cuBLAS Handle

cuBLAS is a stateful library. Instead of using global state, it uses a handle object (cublasHandle_t) to track:

  • Which CUDA stream to use

  • Allocated workspace

  • Algorithm preferences

  • Error state

Explanation

A stateful library is a library that maintains internal state across function calls using a context or handle. This state influences how the functions behave and allows the library to manage things like configuration, resources, or execution context over time.

You can have multiple cuBLAS kernels running in the same program — and even concurrently.

This design allows:

  • Multiple independent cuBLAS contexts

  • Thread safety (you can use different handles in different threads)

  • Better control over performance tuning

The handle is created using cublasCreate, which initializes the cuBLAS library and prepares it for use.

1cublasHandle_t handle;
2cublasCreate(&handle);

The handle must be destroyed when no longer needed to free resources:

1cublasDestroy(handle);

cuBLAS Operations

cublasSgemm is the function for single-precision matrix multiplication. It performs the operation: C = alpha * A * B + beta * C, where:

  • A is an m x n matrix

  • B is an n x k matrix

  • C is an m x k matrix

  • alpha is a scalar multiplier for the product A * B

  • beta is a scalar multiplier for the existing matrix C

Explanation

The leading dimension (ld) is the distance in memory between the start of one column and the start of the next column. For column-major storage (used by cuBLAS), it refers to the number of rows in the matrix. For row-major storage (used by C/C++), it refers to the number of columns, but cuBLAS doesn’t use this directly unless you transpose manually.

 1cublasSgemm(    // Single-precision matrix multiplication
 2    handle,     // cuBLAS handle
 3    CUBLAS_OP_N, // Operation on A (CUBLAS_OP_N for no transpose)
 4    CUBLAS_OP_N, // Operation on B (CUBLAS_OP_N for no transpose)
 5    N,           // Number of rows in A and C
 6    N,           // Number of columns in B and C
 7    N,           // Number of columns in A and rows in B
 8    &alpha,      // Scalar multiplier for A*B
 9    d_A,         // Pointer to matrix A in device memory
10    N,           // Leading dimension of A
11    d_B,         // Pointer to matrix B in device memory
12    N,           // Leading dimension of B
13    &beta,       // Scalar multiplier for C
14    d_C,         // Pointer to matrix C in device memory
15    N);          // Leading dimension of C

Explanation

In cuBLAS don’t have to manually configure thread blocks and grids like you do in raw CUDA kernel launches. cuBLAS internally

  • Inspects the matrix sizes and layout

  • Picks the best kernel and block/thread/grid configuration

  • Launches the kernel using its own internal logic

The final result C is in column-major order, which is the default for cuBLAS. So tp print the result, we can use the following code:

1for (int i = 0; i < N; ++i) {
2    for (int j = 0; j < N; ++j) {
3        std::cout << h_C[IDX2C(i, j, N)] << " ";
4    }
5    std::cout << "\n";
6}

Key Points

  • cuBLAS is a stateful library that uses a handle to manage its state.

  • The handle is created with cublasCreate and destroyed with cublasDestroy.

  • Matrix multiplication is performed using cublasSgemm, which requires specifying the operation type, dimensions, and pointers to the matrices.

  • cuBLAS uses column-major order for matrices, which is different from the row-major order used in C/C++.

  • The leading dimension is important for correctly accessing matrix elements in memory.