Warning

qnn-netron Deprecation Notice: qnn-netron has been deprecated and will be removed in 2.40.

qnn-netron (Beta)

Overview

QNN Netron tool is making model debugging and visualization less daunting. qnn-netron is an extension of the netron graph tool. It provides for easier graph debugging and convenient runtime information. There are currently two key functionalities of the tool:

1. The Visualize section allows customers to view their desired models after using the QNN Converter by importing the JSON representation of the model

2. The Diff section allows customers to run networks of their choosing on different runtimes in order to compare network accuracy and performance

Launching Tool

Dependencies

The QNN netron tool leverages electron JS framework for building GUI frontend and depends on npm/node_js to be available in system. Additionally, python libraries for accuracy analysis are required by backend of tool. A convenient script is available in the QNN SDK to download necessary dependencies for building and running the tool.

# Note: following command should be run as administrator/root to be able to install system libraries
$ sudo bash ${QNN_SDK_ROOT}/bin/check-linux-dependency.sh
$ ${QNN_SDK_ROOT}/bin/check-python-dependency

Launching Application

qnn-netron script is used to be able to launch the QNN Netron application. This script:

1. Clones vanilla netron git project

2. Applies custom patches for enabling Netron for QNN

3. Build the npm project

4. Launches application

$ qnn-netron -h
usage: qnn-netron [-h] [-w <working_dir>]
Script to build and launch QNN Netron tool for visualizing and running analysis on Qnn Models.

Optional argument(s):
 -w <working_dir>                      Location for building QNN Netron tool. Default: current_dir


# To build and run application use
$ qnn-netron -w <my_working_dir>

QNN Netron Visualize Deep Dive

First, the user is prompted to open a JSON file that represents their converted model. This JSON comes from the converter tool. Please refer to this Overview for more details.

Once the file is loaded into the tool, the graph should be displayed in the UI as shown below:

After loading in the model, the user can click on any of the nodes and a side pop-up section will display node information such as the type and name as well as vital parameter information such as inputs and outputs (datatypes, encodings, and shapes)

Netron Diff Customization Deep Dive

Limitations

1. Diff Tool comparison between source framework goldens only works for framework goldens that are spatial first axis order. (NHWC)

2. For usecases where source framework golden is used for comparison, Diff Tool is only tested to work for tensorflow and tensorflow variant frameworks.

In order for the user to open the Diff Customization tool, they can either click file and then “Open Diff…” or on tool startup by clicking “Diff…” as shown below:

Upon launch of the Diff Customization tool, at the top, the user is prompted to select a use case for the tool. There are 3 options to choose from:

For the purposes of this documentation, only inference vs inference will be detailed. The setup procedure for the other use cases is similar. The other two use cases are explained below:

1. Golden vs Inference: Used to test inference run using goldens from a particular ML framework and comparing against the output of a QNN backend

2. Output vs Output: Used to test existing inference results against ML framework goldens OR used to test differences between two existing inference results

3. Inference Vs Inference: Used to test inference between two converted QNN models or the same QNN model on different QNN backends

Inference vs Inference

If this use case is selected, the user is presented with various form fields for the purposes of running two jobs asynchronously with the option of choosing different runtimes for each QNN network being run.

A more detailed view of what the user is prompted is displayed below:

In order to execute the networks, the user has two options:

Running on Host machine

When the Target Device is selected as “host”, the user can only use the CPU as a runtime. In addition, the user can only select “x86_64-linux-clang” as the architecture in this use case.

Running On-Device

When the Target Device is selected as “on-device”, a Device ID is required to connect to the device via adb. Thereafter, the user can select any of the three QNN backend runtimes available (CPU, GPU, or DSPv[68, 69, 73]) and the user can select architecture “aarch64-android”

After choosing the desired target device and runtime configurations, the rest of the fields are explained in detail below:

Note

Users are able to click again and change the location to any of the path fields

Setup Parameters	Configurations to Select
The options for what verifier to run on the outputs of the model are (See Note below table for custom verifier (accuracy + performance) thresholds and see table below for providing custom accuracy verifier hyperparameters):	RtolAtol, AdjustedRtolAtol, TopK, MeanIOU, L1Error, CosineSimilarity, MSE, SQNR
Model JSON	upload <model>_net.json file that was outputted from the QNN converters.
Model Cpp	upload <model>.cpp that was outputted from the QNN converters.
Model Bin	upload <model>.bin that was outputted from the QNN converters.
NDK Path	upload the path to your Android NDK
Devices Engine Path	upload the path to the top-level of the unzipped qnn-sdk
Input List	provide a path to the input file for the model
Save Run Configurations	provide a location where the inference and runtime results from the Diff customization tool will be stored

Note

Users have the option of providing a custom accuracy and performance verifier threshold when running diff. A custom accuracy verifier threshold can be provided for any of the accuracy verifiers. By default the verifier thresholds are 0.01. The custom thresholds can be provided in the text boxes labelled “Accuracy Threshold” and “Perf Threshold”.

Users now have the option to enter accuracy verifier specific hyperparameters inside textboxes. The Default Values are displayed inside the text-boxes and can be customized as per user needs. The table below highlights the hyperparameters for each verifier that can be customized.

Verifier	Hyperparameters
AdjustedRtolAtol	Number of Levels
RtolAtol	Rtol Margin, Atol Margin
Topk	K, Ordered
MeanIOU	Background Classification
L1Error	Multiplier, Scale
CosineSimilarity	Multiplier, Scale
MSE (Mean Square Error)	N/A
SQNR (Signal-To-Noise Ratio)	N/A

Below is an example of what the fields should look like once filled to completion:

After running the Diff Customization tool, the output directories/files should be present in the working directory file path provided in the last field

Results and Outputs:

After pressing the Run button as mentioned above, the visualization of the network should pop-up. Nodes will be highlighted if there are any accuracy and/or performance variations. Clicking on each node will show more information about the accuracy and performance diff information as shown below.

Performance and Accuracy Diff Visualizations:

As seen above, the performance and accuracy diff information is shown under the Diff section of any given node. The color of the node boundary in the viewer represents whether a performance or accuracy error (above the default verifier threshold of 0.01) was reported. For example, in the Conv2d node shown below, there are two boundaries of orange and red indicating that this node has both an accuracy and performance difference across the runs. The FullyConnected node shown only has a yellow boundary indicating that only a performance difference was found.

QNN Netron Diff Navigation

QNN Netron has the ability to locate the first node in the graph with any performance or accuracy diffs. When the user clicks on the next and previous arrows, the visualization of the graph will zoom into the desired node with the first performance or accuracy difference. This makes model debugging much easier for larger models as the user doesn’t have to look for the nodes themselves to find where the network performance and accuracy errors starts to diverge.