QNN GPU QnnMem API Tutorial

Introduction

This tutorial demonstrates the usage of the QnnMem API for the QNN GPU backend. This feature allows for data sharing between processing domains in the QNN GPU backend.

The supported types of shared memory of the QnnMem API for the QNN GPU backend are as follows:

Qnn_MemDescriptor_t Type	QnnGpu_MemType_t Type	Descriptor
QNN_MEM_TYPE_CUSTOM	QNN_GPU_MEM_OPENCL	Each tensor shall be mapped to their own OpenCL buffer One-to-one relationship between OpenCL buffer and memory handle

Note

This tutorial is only focused on the QNN GPU Mem OpenCL buffer usage. There are some prerequisites in the SDK example code not discussed in detail here. Users can refer to the corresponding part in the QNN documentation, or refer to the SampleApp.

SampleApp documentation: Sample App Tutorial

SampleApp code: ${QNN_SDK_ROOT}/examples/QNN/SampleApp

Using QNN_MEM_TYPE_CUSTOM with the QNN API

The following documentation demonstrates the custom memory type feature of the QnnMem API for the QNN GPU backend which allows a user to allocate their own OpenCL buffers to manage and register for input and output tensors. Doing so will achieve zero-copy in which the need to read to and write from the GPU memory to host and vice-versa is mitigated. This will yield inference time metric improvements.

The following is a representation of utilizing OpenCL buffers, where each tensor has its own OpenCL buffer and memory handle.

Below is an example of an implementation of this API. It implies that the user has prior background with OpenCL and its APIs.

GPU OpenCL Buffer Example

// QnnInterface_t is defined in ${QNN_SDK_ROOT}/include/QNN/QnnInterface.h
QnnInterface_t qnnInterface;
// Init qnn interface ......
// See ${QNN_SDK_ROOT}/examples/QNN/SampleApp code

// Qnn_Tensor_t is defined in ${QNN_SDK_ROOT}/include/QNN/QnnTypes.h
Qnn_Tensor_t inputTensor;
// Set up common setting for inputTensor ......
/* There are 2 specific settings for custom QnnMem buffer:
*  1. memType should be QNN_TENSORMEMTYPE_MEMHANDLE; (line 41)
*  2. union member memHandle should be used instead of clientBuf, and it
*     should be set to nullptr. (line 42)
*/


// Allocate some buffer, for example two tensors for the gpu runtime with dimensions {64, 128}
size_t bufferSize = 64 * 128 * sizeof(float);
auto outputTensorBuffer = (float*)malloc(bufferSize);

// Generate OpenCL context
auto clContext = ...;
const cl_mem_flags = CL_MEM_READ_WRITE;
cl_int clStatus;
auto outputTensorCLBuffer = new cl::buffer(*clContext, memFlags, bufferSize, outputTensorBuffer, &clStatus);
if (clStatus != CL_SUCCESS) {
   // Handle error
}

// Fill the info of Qnn_MemDescriptor_t and register the buffer to QNN
// Qnn_MemDescriptor_t is defined in ${QNN_SDK_ROOT}/include/QNN/QnnMem.h
Qnn_MemDescriptor_t memDescriptor = QNN_MEM_DESCRIPTOR_INIT;
memDescriptor.memShape = {inputTensor.rank, inputTensor.dimensions, nullptr};
memDescriptor.dataType = inputTensor.dataType;
memDescriptor.memType = QNN_MEM_TYPE_CUSTOM;

// Fill the info of QnnGpu_MemInfoCustom_t to apply to the Qnn_MemDescriptor_t.
QnnGpu_MemInfoCustom_t customInfo = QNN_GPU_MEM_INFO_CUSTOM_INIT;
customInfo.memType = QNN_GPU_MEM_OPENCL;
customInfo.buffer = reinterpret_cast<QnnGpuMem_Buffer_t>(outputTensorBuffer);
memDescriptor.customInfo = customInfo;
outputTensor.memType = QNN_TENSORMEMTYPE_MEMHANDLE;
outputTensor.memHandle = nullptr;

Qnn_ContextHandle_t context; // Must obtain a QNN context handle before memRegister()
// To obtain QNN context handle:
// For online prepare, refer to ${QNN_SDK_ROOT}/docs/general/sample_app.html#create-context
// For offline prepare, refer to ${QNN_SDK_ROOT}/docs/general/sample_app.html#load-context-from-a-cached-binary
Qnn_MemHandle_t memHandles[1];
auto result = QnnMem_register(context, &memDescriptor, 1u, memHandles);
if (QNN_SUCCESS != result) {
    // handle errors
}

/**
* At this place, the allocation and registration of the OpenCL buffer has been complete.
* On user side, this buffer can be manipulated through outputTensorBuffer;
*/

// Load the input data to outputTensorBuffer ......

// Execute QNN graph with input tensor and output tensor ......

// Get output data for example
auto openCLCommandQueue = ...; // Get cl::CommandQueue instance
auto mappedPtr =
    reinterpret_cast<float*>(openCLCommandQueue->enqueueMapBuffer(*outputTensorBuffer,
                                                                  CL_TRUE,
                                                                  CL_MAP_READ,
                                                                  0,
                                                                  sizeof(bufferSize),
                                                                  nullptr,
                                                                  nullptr,
                                                                  &clStatus);
if (clStatus != CL_SUCCESS) {
    // handle error
}

// Access contents of mappedPtr e.g.
std::vector<float> contents(mappedPtr, mappedPtr + bufferSize);
for (size_t i = 0u; i < contents.size(); i++) {
    // Read data
}

// On completion, unmap the mappedPtr
clStatus = openCLCommandQueue->enqueueUnmapBuffer(*outputTensorBuffer, mappedPtr, nullptr, nullptr);
if (clStatus != CL_SUCCESS) {
    // handle error
}

// Deregister and free all buffers if it's not being used
result = QnnMem_deregister(&tensors.memHandle, 1);
if (QNN_SUCCESS != registRet) {
    // handle errors
}

// deallocate memory