Saver Tutorial: Save execution sequence with Saver and replay on a backend¶

This tutorial describes how to use the QNN Saver backend to capture the execution sequence of QNN APIs. The captured output can then be compiled and replayed on any QNN backend.

Linux¶

Generating saver_output.c¶

saver_output.c is an artifact produced by running a model on the Saver backend. A model in a source framework can be converted into a model.cpp and model.bin using the QNN Converters. Sample model.cpp and model.bin files are located in ${QNN_SDK_ROOT}/examples/QNN/converter/models/.

model.cpp and model.bin are used to create a model.so via qnn-model-lib-generator.

$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-model-lib-generator \
              -c ${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.cpp \
              -b ${QNN_SDK_ROOT}/examples/QNN/converter/models/qnn_model_float.bin \
              -o ${QNN_SDK_ROOT}/examples/QNN/converter/model_libs # This can be any path

This will produce the following artifacts:

${QNN_SDK_ROOT}/examples/QNN/converter/model_libs/aarch64-android/libqnn_model_float.so
${QNN_SDK_ROOT}/examples/QNN/converter/model_libs/x86_64-linux-clang/libqnn_model_float.so

The resulting model.so can be run on the Saver backend using qnn-net-run:

$ cd ${QNN_SDK_ROOT}/examples/QNN/converter/models/
$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-net-run \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnSaver.so \
              --model ${QNN_SDK_ROOT}/examples/QNN/converter/model_libs/x86_64-linux-clang/libqnn_model_float.so \
              --input_list input_list_float.txt

This will produce the following artifacts:

./saver_output/saver_output.c
./saver_output/params.bin

See Tutorial: Converting and executing a CNN model with QNN for more information on converting a model from a source framework into the QNN shared library format.

Alternatively, the following script performs the steps described above and can be used for convenience.

$ cd ${QNN_SDK_ROOT}/examples/QNN/NetRun/linux-x86_64
$ ./linux-qnn-net-run.sh -b saver

This will produce a quantized saver_output.c and params.bin in ${QNN_SDK_ROOT}/examples/QNN/NetRun/linux-x86_64/saver_output/quantized/ for replay on DSP or HTP backends, and a non-quantized saver_output.c and params.bin in ${QNN_SDK_ROOT}/examples/QNN/NetRun/linux-x86_64/saver_output/non_quantized/ for replay on CPU or GPU backends.

Compiling saver_output.c¶

To compile saver_output.c for replay on a backend, use the Makefile located in ${QNN_SDK_ROOT}/examples/QNN/Saver. This makefile can be used to compile saver_output.c for the specified QNN backend on a compatible platform.

Note

Compiling for x86 targets requires clang.
Compiling for Android targets requires Android NDK.
Compiling for DSP/HTP backends on Android targets requires Hexagon SDK.

Refer to Setup to set the appropriate environment variables.

With saver_output.c located in the same directory as Makefile, run make all to compile for all QNN backends on supported targets.

$ cp ./saver_output/saver_output.c ${QNN_SDK_ROOT}/examples/QNN/Saver
$ cp ./saver_output/params.bin ${QNN_SDK_ROOT}/examples/QNN/Saver
$ cd ${QNN_SDK_ROOT}/examples/QNN/Saver
$ make x86 android

This will produce a folder for each target:

./x86_64-linux-clang
./aarch64-android

These folders will contain one executable for each backend available on the target named saver_output_<backend>.

To create executables for all backends on a specific target, run make <target>. Valid options for <target> are x86 or aarch64.

To create an executable for a specific backend on a specific target, run make <backend>_<target>. Valid options for <backend> are cpu or htp for <target>=x86, and cpu, gpu, dsp, hta, or htp for <target>=aarch64.

Replaying saver_output.c on x86¶

To replay on x86 backends, set LD_LIBRARY_PATH to point to the desired QNN backend library and run the executable. Ensure params.bin is in the current working directory.

$ cd ${QNN_SDK_ROOT}/examples/QNN/Saver
$ export QNN_BACKEND=<backend> # where <backend> is QnnCpu or QnnHtp
$ export LD_LIBRARY_PATH=${QNN_SDK_ROOT}/lib/x86_64-linux-clang:$LD_LIBRARY_PATH
$ ./x86_64-linux-clang/saver_output_${QNN_BACKEND} [--logging <verbose,debug,info,warn,error>] # optionally enable logging

Replaying saver_output.c on Android¶

Replay on QNN CPU, QNN HTA, or QNN GPU Backend¶

Make a directory on device, push QNN backend library, saver_output.c executable, and params.bin:

$ export QNN_BACKEND=<backend> # where <backend> is QnnCpu, QnnHta, or QnnGpu
$ adb shell "mkdir /data/local/tmp/saver"
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/lib${QNN_BACKEND}.so /data/local/tmp/saver
$ adb push ${QNN_SDK_ROOT}/examples/QNN/Saver/aarch64-android/saver_output_${QNN_BACKEND} /data/local/tmp/saver
$ adb push ./saver_output/params.bin /data/local/tmp/saver

Set LD_LIBRARY_PATH on device and execute

$ adb shell
$ cd /data/local/tmp/saver
$ export LD_LIBRARY_PATH=/data/local/tmp/saver:$LD_LIBRARY_PATH
$ ./saver_output_${QNN_BACKEND} [-l <verbose,debug,info,warn,error>] # optionally enable logging

Replay on QNN DSP or QNN HTP Backend¶

Make a directory on device, push QNN backend stub/skel, saver_output.c executable, and params.bin:

$ export QNN_BACKEND=<backend>  # where <backend> is QnnDsp or QnnHtp
$ export HEXAGON_ARCH=<v65,v66,v68,v69,v73,v75>  # based on device
$ adb shell "mkdir data/local/tmp/saver"
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/lib${QNN_BACKEND}.so /data/local/tmp/saver
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/lib${QNN_BACKEND}${HEXAGON_ARCH^}Stub.so /data/local/tmp/saver
$ adb push ${QNN_SDK_ROOT}/lib/hexagon-${HEXAGON_ARCH}/unsigned/lib${QNN_BACKEND}${HEXAGON_ARCH^}Skel.so /data/local/tmp/saver
$ adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpPrepare.so /data/local/tmp/saver  # if QNN_BACKEND=QnnHtp
$ adb push ${QNN_SDK_ROOT}/examples/QNN/Saver/aarch64-android/saver_output_${QNN_BACKEND} /data/local/tmp/saver
$ adb push ${QNN_SDK_ROOT}/examples/QNN/Saver/params.bin /data/local/tmp/saver

Set LD_LIBRARY_PATH and ADSP_LIBRARY_PATH on device and execute:

$ adb shell
$ cd /data/local/tmp/saver
$ export LD_LIBRARY_PATH=/data/local/tmp/saver:$LD_LIBRARY_PATH
$ export ADSP_LIBRARY_PATH=/data/local/tmp/saver:$ADSP_LIBRARY_PATH
$ ./saver_output_${QNN_BACKEND} [-l <verbose,debug,info,warn,error>] # optionally enable logging

Windows¶

Setup¶

The tutorial assumes general setup instructions have been followed at Setup.

Please use “x86_x64 Cross Tools Command Prompt for VS 2022” to set QNN_SDK_ROOT:

$ set QNN_SDK_ROOT=\path\to\QNN_SDK_ROOT

Build¶

Please use “x86_x64 Cross Tools Command Prompt for VS 2022” in this step.

Generate Model¶

For CPU, generate a non-quantized model:

For Windows native/x86_x64 PC developers

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ mkdir model && cd model
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_float.cpp" .
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_float.bin" .
$ py -3 "%QNN_SDK_ROOT%\bin\x86_64-windows-msvc\qnn-model-lib-generator" ^
           -c .\qnn_model_float.cpp ^
           -b .\qnn_model_float.bin ^
           -o .\model_libs ^
           -t windows-x86_64

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\model\model_libs\x64\qnn_model_float.dll

For Windows on Snapdragon developers

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ mkdir model && cd model
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_float.cpp" .
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_float.bin" .
$ py -3 "%QNN_SDK_ROOT%\bin\aarch64-windows-msvc\qnn-model-lib-generator" ^
           -c .\qnn_model_float.cpp ^
           -b .\qnn_model_float.bin ^
           -o .\model_libs ^
           -t windows-aarch64

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\model\model_libs\ARM64\qnn_model_float.dll

For DSP and HTP, generate a quantized model:

For Windows native/x86_x64 PC developers

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ mkdir model && cd model
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_8bit_quantized.cpp" .
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_8bit_quantized.bin" .
$ py -3 "%QNN_SDK_ROOT%\bin\x86_64-windows-msvc\qnn-model-lib-generator" ^
           -c .\qnn_model_8bit_quantized.cpp ^
           -b .\qnn_model_8bit_quantized.bin ^
           -o .\model_libs ^
           -t windows-x86_64

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\model\model_libs\x64\qnn_model_8bit_quantized.dll

For Windows on Snapdragon developers

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ mkdir model && cd model
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_8bit_quantized.cpp" .
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\qnn_model_8bit_quantized.bin" .
$ py -3 "%QNN_SDK_ROOT%\bin\aarch64-windows-msvc\qnn-model-lib-generator" ^
           -c .\qnn_model_8bit_quantized.cpp ^
           -b .\qnn_model_8bit_quantized.bin ^
           -o .\model_libs ^
           -t windows-aarch64

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\model\model_libs\ARM64\qnn_model_8bit_quantized.dll

Generate saver_output.c and params.bin¶

Please check the path of <model>.dll and replace in the below commands:

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ mkdir source && cd source
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\input_data_float" .\input_data_float /i
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\converter\models\input_list_float.txt" .
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\Saver\model\model_libs\x64\<model>.dll" .
$ "%QNN_SDK_ROOT%\bin\x86_64-windows-msvc\qnn-net-run.exe" ^
           --backend "%QNN_SDK_ROOT%\lib\x86_64-windows-msvc\QnnSaver.dll" ^
           --model <model>.dll ^
           --input_list .\input_list_float.txt

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\saver_output.c
%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\params.bin

Build saver_output.c¶

Please choose one target between [x64, ARM64] as the parameter of “-A” config. Executables for all backends on the target named saver_output_<backend> will be generated.

$ cd "%QNN_SDK_ROOT%\examples\QNN\Saver"
$ xcopy "%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\saver_output.c" .
$ mkdir build && cd build
$ cmake -S ../ -B ./ -T ClangCL -A [x64, ARM64]
$ cmake --build ./ --config Release

To generate an executable for a specific backend, please use below command to build. Valid options for <backend> are cpu for <target>=x64, and cpu, dsp, or htp for <target>=ARM64.

$ cmake --build ./ --config Release --target [cpu, dsp, htp]

After executing commands from above, you should be able to see:

%QNN_SDK_ROOT%\examples\QNN\Saver\build\Release\saver_output_<backend>.exe

Run¶

Please use “Command Prompt” to execute, and ensure the params.bin is the same file generated in build.

For CPU¶

Run on x86.

Please push below files to a folder:

%QNN_SDK_ROOT%\examples\QNN\Saver\build\Release\saver_output_QnnCpu.exe
%QNN_SDK_ROOT%\lib\x86_64-windows-msvc\QnnCpu.dll
%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\params.bin

To execute:

$ .\saver_output_QnnCpu.exe [--logging <verbose,debug,info,warn,error>] # optionally enable logging

Run on aarch64.

Please push below files to device:

%QNN_SDK_ROOT%\examples\QNN\Saver\build\Release\saver_output_QnnCpu.exe

%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnCpu.dll

%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\params.bin

To execute:

$ .\saver_output_QnnCpu.exe [--logging <verbose,debug,info,warn,error>] # optionally enable logging

For DSP¶

Run on aarch64.

Please push below files to device:

%QNN_SDK_ROOT%\examples\QNN\Saver\build\Release\saver_output_QnnDsp.exe
%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnDsp.dll
%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnDspV66Stub.dll
%QNN_SDK_ROOT%\lib\hexagon-v66\unsigned\libQnnDspV66Skel.so
%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\params.bin

To execute:

$ .\saver_output_QnnDsp.exe [--logging <verbose,debug,info,warn,error>] # optionally enable logging

For HTP¶

Run on aarch64.

Please push below files to device:

%QNN_SDK_ROOT%\examples\QNN\Saver\build\Release\saver_output_QnnHtp.exe
%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnHtp.dll
%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnHtpPrepare.dll
%QNN_SDK_ROOT%\lib\aarch64-windows-msvc\QnnHtp<Hexagon Architecture>Stub.dll
%QNN_SDK_ROOT%\lib\hexagon-v68\unsigned\libQnnHtp<Hexagon Architecture>Skel.so
%QNN_SDK_ROOT%\examples\QNN\Saver\source\saver_output\params.bin

To execute:

$ .\saver_output_QnnHtp.exe [--logging <verbose,debug,info,warn,error>] # optionally enable logging