Tutorial: Executing a shallow model using custom op package

The following tutorial will demonstrate the use of the QNN API and QNN tools with an example Op Package. The simulated workflow will use a shallow network as the model and Relu as an example operation.

The tutorial will use an example shallow network model, and execute the model using the C++ application qnn-net-run. The execution will show usage on the CPU, GPU, HTP and DSP backends on both host (for CPU) and device.

The sections of the tutorial are as follows:

  1. Tutorial Setup

  2. Building Example Op Package

  3. Building Example Model

  4. Executing Example Model

Tutorial Setup

The tutorial assumes general setup instructions have been followed at Setup.

Building Example Op Package

Sources for an example Op Package, containing the Relu operation, are present for the CPU, GPU, DSP, and HTP backend. Each backend has different requirements for building the Op Package consumable by qnn-net-run.

Compiling for CPU Backend

The CPU Backend Example Op Package is located at:

${QNN_SDK_ROOT}/examples/QNN/OpPackage/CPU

Building the example Op Package for CPU by default builds for Linux x86-64 and Android architectures, and has a dependency on the Android NDK Toolchain. Ensure that ANDROID_NDK_ROOT is set. See Setup for more information.

To build the example Op Package for CPU:

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/CPU
$ make

This will produce the artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/CPU/libs/<target>/libQnnCpuOpPackageExample.so

Note

The desired architecture can be specified using provided make targets: cpu_aarch64-android, cpu_x86

Note

To compile for only Linux x86-64 architectures replace the command “make” with “make cpu_x86”. To compile for only Android architectures replace the command “make” with “make cpu_android”.

Compiling for GPU Backend

The GPU Backend example Op Package is located at:

${QNN_SDK_ROOT}/examples/QNN/OpPackage/GPU

Compilation of the GPU Op Package has a dependency on the Android NDK Toolchain. Ensure that ANDROID_NDK_ROOT is set. See Setup for more information.

To compile the example GPU Op Package

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/GPU
$ make

This will produce artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/GPU/libs/<target>/libQnnGpuOpPackageExample.so

Compiling for HTP Backend

The HTP Backend Example Op Package is located at:

${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP

HTP Emulation x86 Compilation

Compiling the example Op Package for the HTP emulation has dependencies on clang++ and the Hexagon SDK.

$ export X86_CXX=<path-to-clang++>
$ export HEXAGON_SDK_ROOT=<path-to-hexagon-sdk>
$ export QNN_INCLUDE=${QNN_SDK_ROOT}/include/QNN

To compile the example HTP Emulation Op Package for use on the Linux x86-64 architecture:

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP
$ make htp_x86

This will produce the artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/x86_64-linux-clang/libQnnHtpOpPackageExample.so

HTP Hexagon V68 Compilation

Compiling the example Op Package for the HTP Hexagon V68 has dependencies on Hexagon SDK.

$ export HEXAGON_SDK_ROOT=<path-to-hexagon-sdk>
$ export QNN_INCLUDE=${QNN_SDK_ROOT}/include/QNN

To compile the example HTP Op Package for use on the Hexagon V68 architecture:

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP
$ make htp_v68

This will produce the artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/hexagon-v68/libQnnHtpOpPackageExample.so

HTP ARM Compilation

Compiling the example Op Package for the HTP Hexagon V68 has dependencies on Android NDK.

$ export ANDROID_NDK_ROOT=<path-to-android-ndk>
$ export QNN_INCLUDE=${QNN_SDK_ROOT}/include/QNN

To compile the example HTP Op Package for use on the Android ARM architecture:

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP
$ make htp_aarch64

This will produce the artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/aarch64-android/libQnnHtpOpPackageExample.so

Compiling for DSP Backend

The DSP Backend Example Op Package is located at:

${QNN_SDK_ROOT}/examples/QNN/OpPackage/DSP

Compiling the example Op Package for the DSP has dependencies on Hexagon SDK.

$ export HEXAGON_SDK_ROOT=<path-to-hexagon-sdk>
$ export QNN_SDK_ROOT=${QNN_SDK_ROOT}

To compile the example DSP Op Package:

$ cd ${QNN_SDK_ROOT}/examples/QNN/OpPackage/DSP
$ make

This will produce the artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/OpPackage/DSP/build/DSP/libQnnDspOpPackageExample.so

Building Example Model

The shallow model in this tutorial is post-use of the QNN Converter. Information on the QNN converter can be found on the Tools page.

The model is located at:

${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.cpp

Additionally, for the HTP and DSP backend there is a quantized model

${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_8bit_quantized.cpp

The remainder of the tutorial will use qnn_model_float.cpp. For the quantized model replace qnn_model_float.cpp with qnn_model_8bit_quantized.cpp and qnn_model_float.bin with qnn_model_8bit_quantized.bin.

In order to use the example Op Package, instead of QNN’s default Op Package, the model must be manually edited to refer to the example. The process is outlined here:

$ vim ${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.cpp

Before:

242   VALIDATE(convReluModel.addNode(
243                QNN_OPCONFIG_VERSION_1,                              // Op_Config_t Version
244                "InceptionV3_InceptionV3_Conv2d_1a_3x3_Relu",        // Node Name
245                "qti.aisw",                                          // Package Name Default

After:

242   VALIDATE(convReluModel.addNode(
243                QNN_OPCONFIG_VERSION_1,                             // Op_Config_t Version
244                "InceptionV3_InceptionV3_Conv2d_1a_3x3_Relu",       // Node Name
245                "examples.OpPackage",                               // Package Name To Change

After the appropriate Op Package name has been populated, the model can be built for the desired target using qnn-model-lib-generator. Before generating the model, ensure ANDROID_NDK_ROOT is set. See Setup for more information.

To build the model use:

$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-model-lib-generator \
  -c ${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.cpp \
  -b ${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.bin \
  -o ${QNN_SDK_ROOT}/examples/QNN/example_libs # This can be any path

This will produce the following artifact(s):

  • ${QNN_SDK_ROOT}/examples/QNN/example_libs/<target>/libqnn_model_float.so

Note

Any output path can be used with the -o flag. The remainder of this tutorial will use ${QNN_SDK_ROOT}/examples/QNN/example_libs to refer to the directory containing model libraries.

CPU Backend Execution

Running CPU Backend on Linux x86

With the libraries discoverable, qnn-net-run is used with the following:

$ cd ${QNN_SDK_ROOT}/examples/QNN/converter/models # access input data
$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-net-run \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnCpu.so \
              --model ${QNN_SDK_ROOT}/examples/QNN/example_libs/x86_64-linux-clang/libqnn_model_float.so \
              --input_list ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt \
              --op_packages ${QNN_SDK_ROOT}/examples/QNN/OpPackage/CPU/libs/x86_64-linux-clang/libQnnCpuOpPackageExample.so:QnnOpPackage_interfaceProvider

Outputs from the run will be located at the default ./output directory.

Note

If full paths are not given to qnn-net-run, all libraries must be added to LD_LIBRARY_PATH and be discoverable by the system library loader.

Running CPU Backend on Android

Running the CPU Backend on an Android target is largely similar to running on the Linux x86 target.

First, create a directory for the example on device:

# make oppackage if necessary
$ adb shell "mkdir /data/local/tmp/oppackage"
$ adb shell "mkdir /data/local/tmp/oppackage/CPU"

Now push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnCpu.so /data/local/tmp/oppackage/CPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/CPU/libs/aarch64-android/libQnnCpuOpPackageExample.so /data/local/tmp/oppackage/CPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/*.so /data/local/tmp/oppackage/CPU

Now push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/oppackage/CPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/oppackage/CPU

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/oppackage/CPU

Now set up the environment on device:

$ adb shell
$ cd /data/local/tmp/oppackage/CPU
$ export LD_LIBRARY_PATH=/data/local/tmp/oppackage/CPU

Finally, use qnn-net-run with the following:

$ ./qnn-net-run --backend libQnnCpu.so --model libqnn_model_float.so --input_list input_list_float.txt --op_packages libQnnCpuOpPackageExample.so:QnnOpPackage_interfaceProvider

Outputs from the run will be located at the default ./output directory.

GPU Backend Execution

Running GPU Backend on Android

Running the GPU Backend on an Android target is largely similar to running CPU Backend on Android.

First, create a directory for the example on device:

# make oppackage if necessary
$ adb shell "mkdir /data/local/tmp/oppackage"
$ adb shell "mkdir /data/local/tmp/oppackage/GPU"

Now push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnGpu.so /data/local/tmp/oppackage/GPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/GPU/libs/aarch64-android/libQnnGpuOpPackageExample.so /data/local/tmp/oppackage/GPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/*.so /data/local/tmp/oppackage/GPU

Now push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/oppackage/GPU
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/oppackage/GPU

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/oppackage/GPU

Now set up the environment on device:

$ adb shell
$ cd /data/local/tmp/oppackage/GPU
$ export LD_LIBRARY_PATH=/data/local/tmp/oppackage/GPU

Finally, use qnn-net-run with the following:

$ ./qnn-net-run --backend libQnnGpu.so --model libqnn_model_float.so --input_list input_list_float.txt --op_packages libQnnGpuOpPackageExample.so:QnnOpPackage_interfaceProvider

Outputs from the run will be located at the default ./output directory.

HTP Backend Execution

Running HTP Emulation Backend on Linux x86

With the appropriate libraries compiled, qnn-net-run is used with the following:

$ cd ${QNN_SDK_ROOT}/examples/QNN/converter/models
$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-net-run \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnHtp.so \
              --model ${QNN_SDK_ROOT}/examples/QNN/example_libs/x86_64-linux-clang/libqnn_model_8bit_quantized.so \
              --input_list ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt \
              --op_packages ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/x86_64-linux-clang/libQnnHtpOpPackageExample.so:exampleInterfaceProvider

Outputs from the run will be located at the default ./output directory.

Note

If full paths are not given to qnn-net-run, all libraries must be added to LD_LIBRARY_PATH and be discoverable by the system library loader.

Running HTP Backend on Android using offline prepared graph

Running the HTP Backend on an Android target is largely similar to running CPU and GPU Backend. In this tutorial, we first prepare the graph on x86 host, then pass the serialized context binary to the device HTP Backend to execute. To run the graph on the HTP backend, the aarch64 version of the OpPackage is needed in addition to the hexagon-v68 version.

Additionally, running the HTP on device can be done with the generation of a serialized context. This serialized context can be initialized more efficiently by HTP compared to the original libqnn_model_8bit_quantized.so model. To generate the context, run:

$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-context-binary-generator \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnHtp.so \
              --model ${QNN_SDK_ROOT}/examples/QNN/example_libs/x86_64-linux-clang/libqnn_model_8bit_quantized.so \
              --op_packages ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/x86_64-linux-clang/libQnnHtpOpPackageExample.so:exampleInterfaceProvider
              --binary_file qnn_model_8bit_quantized.serialized

This creates the context at:

  • ./output/qnn_model_8bit_quantized.serialized.bin

First, create a directory for the example on device:

# make oppackage if necessary
$ adb shell "mkdir /data/local/tmp/oppackage"
$ adb shell "mkdir /data/local/tmp/oppackage/HTP"

Now push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtp.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpPrepare.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV68Stub.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/hexagon-v68/libQnnHtpOpPackageExample.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/aarch64-android/libQnnHtpOpPackageExample.so /data/local/tmp/oppackage/HTP/libQnnHtpOpPackageExample_Cpu.so
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/*.so /data/local/tmp/oppackage/HTP
$ adb push ./output/qnn_model_8bit_quantized.serialized.bin /data/local/tmp/oppackage/HTP
$ # Additionally, the HTP requires Hexagon specific libraries
$ adb push ${QNN_SDK_ROOT}/lib/hexagon-v68/unsigned/libQnnHtpV68Skel.so /data/local/tmp/oppackage/HTP

Now push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/oppackage/HTP

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/oppackage/HTP

Now set up the environment on device:

$ adb shell
$ cd /data/local/tmp/oppackage/HTP
$ export LD_LIBRARY_PATH=/data/local/tmp/oppackage/HTP
$ export ADSP_LIBRARY_PATH="/data/local/tmp/oppackage/HTP"

Finally, use qnn-net-run with the following:

$ ./qnn-net-run --backend libQnnHtp.so --retrieve_context qnn_model_8bit_quantized.serialized.bin --input_list input_list_float.txt --op_packages libQnnHtpOpPackageExample_Cpu.so:exampleInterfaceProvider:CPU,libQnnHtpOpPackageExample_Htp.so:exampleInterfaceProvider:HTP

In this case two op packages are passed to qnn-net-run. The first one, libQnnHtpOpPackageExample_Cpu.so, is the ARM aarch64 build of the op package. The second op package, libQnnHtpOpPackageExample_Htp.so, is the hexagon v68 build of the same op package. “:CPU” and “:HTP” are the optional target parameters specifying the target platforms on which the backend must register the op packages. The “CPU” target indicates that the op package is compiled for CPU (ARM aarch64). The “HTP” target indicates that the op package is compiled for HTP on-device.

Running HTP Backend on Android using on-device prepared graph

In this tutorial we prepare the model on the Android ARM (CPU) side.

First, create a directory for the example on device:

# make oppackage if necessary
$ adb shell "mkdir /data/local/tmp/oppackage"
$ adb shell "mkdir /data/local/tmp/oppackage/HTP"

Now push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtp.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpPrepare.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV68Stub.so /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/hexagon-v68/libQnnHtpOpPackageExample.so /data/local/tmp/oppackage/HTP/libQnnHtpOpPackageExample_Htp.so
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/HTP/build/aarch64-android/libQnnHtpOpPackageExample.so /data/local/tmp/oppackage/HTP/libQnnHtpOpPackageExample_Cpu.so
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/*.so /data/local/tmp/oppackage/HTP
$ # Additionally, the HTP requires Hexagon specific libraries
$ adb push ${QNN_SDK_ROOT}/lib/hexagon-v68/unsigned/libQnnHtpV68Skel.so /data/local/tmp/oppackage/HTP

Now push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/oppackage/HTP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/oppackage/HTP

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/oppackage/HTP

Now set up the environment on device:

$ adb shell
$ cd /data/local/tmp/oppackage/HTP
$ export LD_LIBRARY_PATH=/data/local/tmp/oppackage/HTP
$ export ADSP_LIBRARY_PATH="/data/local/tmp/oppackage/HTP"

Finally, use qnn-net-run with the following:

$ ./qnn-net-run --backend libQnnHtp.so --model libqnn_model_8bit_quantized.so --input_list input_list_float.txt --op_packages libQnnHtpOpPackageExample_Cpu.so:exampleInterfaceProvider:CPU,libQnnHtpOpPackageExample_Htp.so:exampleInterfaceProvider:HTP

In this case two op packages are passed to qnn-net-run. The first one, libQnnHtpOpPackageExample_Cpu.so, is the ARM aarch64 build of the op package. The second op package, libQnnHtpOpPackageExample_Htp.so, is the hexagon v68 build of the same op package. “:CPU” and “:HTP” are the optional target parameters specifying the target platforms on which the backend must register the op packages. The “CPU” target indicates that the op package is compiled for CPU (ARM aarch64). The “HTP” target indicates that the op package is compiled for HTP on-device.

DSP Backend Execution

Running DSP Backend on Android

Running the DSP Backend on an Android target is largely similar to running HTP Backend on the Android. And the following assumes running on Hexagon DSP v66 device.

First, create a directory for the example on device:

# make oppackage if necessary
$ adb shell "mkdir /data/local/tmp/oppackage"
$ adb shell "mkdir /data/local/tmp/oppackage/DSP"

Now push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnDsp.so /data/local/tmp/oppackage/DSP
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnDspV66Stub.so /data/local/tmp/oppackage/DSP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/OpPackage/DSP/build/DSP/libQnnDspOpPackageExample.so /data/local/tmp/oppackage/DSP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/*.so /data/local/tmp/oppackage/DSP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/example_libs/aarch64-android/libqnn_model_8bit_quantized.so /data/local/tmp/oppackage/DSP
$ # Additionally, the DSP requires Hexagon specific libraries
$ adb push ${QNN_SDK_ROOT}/lib/hexagon-v66/unsigned/libQnnDspV66Skel.so /data/local/tmp/oppackage/DSP

Now push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/oppackage/DSP
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/oppackage/DSP

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/oppackage/DSP

Now set up the environment on device:

$ adb shell
$ cd /data/local/tmp/oppackage/DSP
$ export VENDOR_LIB=/vendor/lib/ # /vendor/lib64/ if aarch64
$ export LD_LIBRARY_PATH=/data/local/tmp/oppackage/DSP:/vendor/dsp/cdsp:$VENDOR_LIB
$ export ADSP_LIBRARY_PATH="/data/local/tmp/oppackage/DSP;/vendor/dsp/cdsp;/vendor/lib/rfsa/adsp;/system/lib/rfsa/adsp;/dsp"

Finally, use qnn-net-run with the following:

$ ./qnn-net-run --backend libQnnDsp.so --model libqnn_model_8bit_quantized.so --input_list input_list_float.txt --op_packages libQnnDspOpPackageExample.so:ExampleReluPackageInterfaceProvider