Tools

This page describes the various tools included in the Qualcomm® AI Engine Direct Delegate and their features.

• qtld-net-run

qtld-net-run

The qtld-net-run tool is used to perform inference with a model using an input list, while saving the model’s output tensors to disk. This tool can run using TFLite’s own CPU runtime or it may utilize the Qualcomm® AI Engine Direct Delegate and perform inference using a specific backend.

Note

Please check qtld-net-run –help for latest options.

An example input file list is shown below, demonstrating the format needed for inputs to qtld-net-run:

inputs/1.raw
inputs/2.raw
inputs/3.raw
inputs/4.raw
inputs/5.raw

This input list file showcases relative file paths (from where qtld-net-run is being executed) to the preprocessed input tensors on the device. In this example, there are 5 preprocessed input tensors within a folder called inputs in the same directory as the text file list.

Below is another example of an input list for a model with two inputs.

inputs/1_a.raw input/1_b.raw
inputs/2_a.raw input/2_b.raw
inputs/3_a.raw input/3_b.raw
inputs/4_a.raw input/4_b.raw
inputs/5_a.raw input/5_b.raw

Like above, each line coincides to one inference. However, the space separated list indicates that two input files should be loaded and passed to the model. The order of the input files, from left to right, should be the same as the order of the input tensors in the TFLite model.

This input list file showcases relative file paths (from where qtld-net-run is being executed) to the preprocessed input tensors on the device. In this example, there are 5 preprocessed input tensors within a folder called inputs in the same directory as the text file list.

Another way to provide the relative file path in the input list is by specifying the input layer name. This can be specified with the below format:

<input_layer_name>:=<input_layer_path>[<space><input_layer_name>:=<input_layer_path>]
[<input_layer_name>:=<input_layer_path>[<space><input_layer_name>:=<input_layer_path>]]
...

Below is an example containing 3 sets of inputs with layer names “Input_1” and “Input_2”, and files located in the relative path “inputs/” and “input/”:

Input_1:=inputs/1_a.raw Input_2:=input/1_b.raw
Input_1:=inputs/2_a.raw Input_2:=input/2_b.raw
Input_1:=inputs/3_a.raw Input_2:=input/3_b.raw

Below is an example of how to run inference on a model through TFLite CPU Runtime:

$ adb shell '/data/local/tmp/qnn_delegate/qtld-net-run \
    --model <path to model on device> \
    --input <path to input file list> \
    --output <output directory path on device>'

Below is an example of how to run inference on a model through a Qualcomm® AI Engine Direct backend with Qualcomm® AI Engine Direct Delegate. Options regarding profiling is optional, default will be no profiling. Note that using the delegate requires setting environment variables like $LD_LIBRARY_PATH or $ADSP_LIBRARY_PATH, as mentioned in setup:On Device Environment Setup.

$ adb shell 'LD_LIBRARY_PATH=/data/local/tmp/qnn_delegate/:$LD_LIBRARY_PATH \
    ADSP_LIBRARY_PATH="/data/local/tmp/qnn_delegate/" \
    /data/local/tmp/qnn_delegate/qtld-net-run \
    --model <path to model on device> \
    --input <path to input file list> \
    --output <output directory path on device> \
    --backend <backend type> \
    --profiling <profiling> \
    --profiling_output_dir <output binary path> \
    --library_path <path to QNN Library on device>'