Accuracy Debugger (Experimental)¶
Dependencies
The Accuracy Debugger depends on the setup outlined in Setup. In particular, the following are required:
Platform dependencies are need to be met as per Platform Dependencies
The desired ML frameworks need to be installed. Accuracy debugger is verified to work with the ML framework versions mentioned at Environment Setup
The following environment variables are used inside this guide (User may change the following paths depending on their needs):
RESOURCESPATH = {Path to the directory where all models and input files reside}
PROJECTREPOPATH = {Path to your accuracy debugger project directory}
Supported models
The snpe-accuracy-debugger currently supports ONNX, TFLite, and Tensorflow 1.x models.
Overview
The accuracy-debugger tool finds inaccuracies in a neural-network at the layer level. The tool compares the golden outputs produced by running a model through a specific ML framework (ie. Tensorflow, Onnx, TFlite) with the results produced by running the same model through Qualcomm’s SNPE Inference Engine. The inference engine can be run on a variety of computing mediums including GPU, CPU and DSP.
The following features are available in Accuracy Debugger. Each feature can be run with its corresponding option; for example, snpe-accuracy-debugger --{option}.
snpe-accuracy-debugger –framework_runner This feature uses frameworks ie. tensorflow, tflite and onnx to run the model to get intermediate outputs. Note: The argument –framewok_diagnosis has been replaced by –framework_runner. –framework_diagnosis will be deprecated in the future release.
snpe-accuracy-debugger –inference_engine This feature uses SNPE engine to run the models to get intermediate outputs.
snpe-accuracy-debugger –verification This feature compares the output generated by framework runner and inference engine features using the verifiers like CosineSimilarity, RtolAtol, etc.
snpe-accuracy-debugger –compare_encodings This feature extracts encodings from a given SNPE DLC, compares them with the given AIMET encodings, and outputs an Excel sheet highlighting mismatches.
snpe-accuracy-debugger –tensor_inspection This feature compares given target outputs with reference outputs.
snpe-accuracy-debugger –quant_checker This feature is used to analyze the activations, weights and biases of all the possible quantization options available in the snpe-converters for each subsequent layer of a single model file or a directory of a model.
- Tip:
You can use –help after the bin commands to see what other options (required or optional) you can add.
If no option is provided, Accuracy Debugger runs framework_runner, inference_engine, and verification sequentially.
Below are the instructons for running the Accuracy Debugger
Framework Runner
The Framework Runner feature is designed to run models with different machine learning frameworks (i.e. Tensorflow, etc). A selected model is run with a specific ML framework. Golden outputs are produced for future comparison with inference results from the Inference Engine step.
Usage
usage: snpe-accuracy-debugger --framework_runner [-h]
-f FRAMEWORK [FRAMEWORK ...]
-m MODEL_PATH
-i INPUT_TENSOR [INPUT_TENSOR ...]
-o OUTPUT_TENSOR
[-w WORKING_DIR]
[--output_dirname OUTPUT_DIRNAME]
[-v]
Script to generate intermediate tensors from a ML Framework.
optional arguments:
-h, --help show this help message and exit
required arguments:
-f FRAMEWORK [FRAMEWORK ...], --framework FRAMEWORK [FRAMEWORK ...]
Framework type and version, version is optional. Currently
supported frameworks are ["tensorflow","onnx","tflite"] case
insensitive but spelling sensitive
-m MODEL_PATH, --model_path MODEL_PATH
Path to the model file(s).
-i INPUT_TENSOR [INPUT_TENSOR ...], --input_tensor INPUT_TENSOR [INPUT_TENSOR ...]
The name, dimensions, raw data, and optionally data
type of the network input tensor(s) specifiedin the
format "input_name" comma-separated-dimensions path-
to-raw-file, for example: "data" 1,224,224,3 data.raw
float32. Note that the quotes should always be
included in order to handle special characters,
spaces, etc. For multiple inputs specify multiple
--input_tensor on the command line like:
--input_tensor "data1" 1,224,224,3 data1.raw
--input_tensor "data2" 1,50,100,3 data2.raw float32.
-o OUTPUT_TENSOR, --output_tensor OUTPUT_TENSOR
Name of the graph's specified output tensor(s).
optional arguments:
-w WORKING_DIR, --working_dir WORKING_DIR
Working directory for the framework_runner to store
temporary files. Creates a new directory if the
specified working directory does not exist
--output_dirname OUTPUT_DIRNAME
output directory name for the framework_runner to
store temporary files under
<working_dir>/framework_runner. Creates a new
directory if the specified working directory does not
exist
-v, --verbose Verbose printing
Please note: All the command line arguments should either be provided through command line or through the config file. They will not override those in the config file if there is overlap.
Sample Commands
snpe-accuracy-debugger \
--framework_runner \
--framework tensorflow \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0
snpe-accuracy-debugger \
--framework_runner \
--framework onnx \
--model_path $RESOURCESPATH/samples/dlv3onnx/dlv3plus_mbnet_513-513_op9_mod_basic.onnx \
--input_tensor Input 1,3,513,513 $RESOURCESPATH/samples/dlv3onnx/data/00000_1_3_513_513.raw \
--output_tensor Output
TIP:
a working_directory, if not otherwise specified, is generated wherever you are calling the script from; it is recommended to call all scripts from the same directory so all your outputs and results are stored under the same directory without having outputs everywhere.
for tensorflow it is sometimes necessary to add the :0 after the input and output node name to signify the index of the node. Notice that the :0 is dropped for onnx models.
Output
The program also creates a folder named latest found in working_directory/framework_runner which is symbolic linked to the most recently generated directory. In the example below, latest will have data that is symbol linked to the data in the most recent directory YYYY-MM-DD_HH:mm:ss. User may choose to override the directory name by passing it to –output_dirname (i.e. –output_dirname myTest1Ouput).
The float data produced by the Framework Runner step offers precise reference material for the Verification component to diagnose the accuracy of the network generated by the Inference Engine. Unless a path is otherwise specified, the Accuracy Debugger will create directories within the working_directory/framework_runner directory found in the current working directory. The directories will be named with the date and time of the program’s execution, and contain tensor data. Depending on the tensor naming convention of the model, there may be numerous sub-directories within the new directory. This occurs when tensor names include a slash “/”. For example, for the tensor names ‘inception_3a/1x1/bn/sc’, ‘inception_3a/1x1/bn/sc_internal’ and ‘inception_3a/1x1/bn’, subdirectories will be generated.
The figure above shows a sample output from one of the framework_runner runs. Framework Runner basically runs inference at each op in the network. InceptionV3 and Logits contain the outputs of each layer before the last layer. Each output directory contains the .raw files corresponding to each node. Every raw file that can be seen is the output of an op. The outputs of the final layer are saved inside Predictions directory. The file framework_runner_options.json contains all the options used to run this feature.
Inference Engine
The Inference Engine feature is designed to find the outputs for a SNPE SDK compatible model. The output produced by this step can be compared with the Golden outputs produced by the framework runner step.
Usage
usage: snpe-accuracy-debugger --inference_engine [-h]
[--stage {source,converted,compiled}]
-r {cpu,gpu,dsp,dspv68,dspv69,dspv73,dspv75}
-p ENGINE_PATH
-a {aarch64-android,aarch64-android-clang6.0,aarch64-android-clang8.0,x86_64-linux-clang,x86_64-windows-msvc,wos}
-l INPUT_LIST
[-i INPUT_TENSOR [INPUT_TENSOR ...]]
[-o OUTPUT_TENSOR] [-m MODEL_PATH]
[-f FRAMEWORK [FRAMEWORK ...]]
[--static_model STATIC_MODEL]
[--deviceId DEVICEID] [-v]
[--host_device {x86,x86_64-windows-msvc,wos}] [-w WORKING_DIR]
[--output_dirname OUTPUT_DIRNAME]
[--debug_mode_off] [-bbw {8,32}]
[-abw {8,16}] [-wbw {8,16}]
[-nif] [-nof] [-qo QUANTIZATION_OVERRIDES]
[--golden_dir_for_mapping GOLDEN_DIR_FOR_MAPPING]
[-mn MODEL_NAME] [--args_config ARGS_CONFIG]
[--print_version PRINT_VERSION]
[--perf_profile
{low_balanced,balanced,high_performance,sustained_high_performance,burst,low_power_saver,power_saver,high_power_saver,system_settings}]
[--offline_prepare]
[--extra_converter_args EXTRA_CONVERTER_ARGS]
[--extra_runtime_args EXTRA_RUNTIME_ARGS]
[--profiling_level {off,basic,moderate,detailed,linting}]
[--extra_quantizer_args EXTRA_QUANTIZER_ARGS]
[--fine_grain_mode FINE_GRAIN_MODE]
[--no_weight_quantization]
[--use_symmetric_quantize_weights]
[--use_enhanced_quantizer]
[--htp_socs HTP_SOCS]
[--use_adjusted_weights_quantizer]
[--override_params]
[--precision {int8,fp16,fp32}]
Script to run SNPE inference engine.
optional arguments:
-h, --help show this help message and exit
Core Arguments:
--stage {source,converted,compiled}
Specifies the starting stage in the Accuracy Debugger
pipeline. Source: starting with a source framework model [default].
Compiled: starting with a model's .so binary
-r {cpu,gpu,dsp,dspv68,dspv69,dspv73,dspv75}, --runtime {cpu,gpu,dsp,dspv68,dspv69,dspv73,dspv75}
Runtime to be used.
-a {aarch64-android,aarch64-android-clang6.0,aarch64-android-clang8.0,x86_64-linux-clang,x86_64-windows-msvc,wos}, --architecture {aarch64-android,aarch64-android-clang6.0,aarch64-android-clang8.0,x86_64-linux-clang,x86_64-windows-msvc,wos}
Name of the architecture to use for inference engine.
-l INPUT_LIST, --input_list INPUT_LIST
Path to the input list text.
Arguments required for SOURCE stage:
-i INPUT_TENSOR [INPUT_TENSOR ...], --input_tensor INPUT_TENSOR [INPUT_TENSOR ...]
The name, dimension, and raw data of the network input
tensor(s) specified in the format "input_name" comma-
separated-dimensions path-to-raw-file, for example:
"data" 1,224,224,3 data.raw. Note that the quotes
should always be included in order to handle special
characters, spaces, etc. For multiple inputs specify
multiple --input_tensor on the command line like:
--input_tensor "data1" 1,224,224,3 data1.raw
--input_tensor "data2" 1,50,100,3 data2.raw.
-o OUTPUT_TENSOR, --output_tensor OUTPUT_TENSOR
Name of the graph's output tensor(s).
-m MODEL_PATH, --model_path MODEL_PATH
Path to the model file(s).
-f FRAMEWORK [FRAMEWORK ...], --framework FRAMEWORK [FRAMEWORK ...]
Framework type to be used, followed optionally by
framework version.
Arguments required for CONVERTED or COMPILED stage:
--static_model STATIC_MODEL
Path to the converted model.
Optional Arguments:
--precision {int8,fp16,fp32}
Choose the precision. Default is int8.
Note: This option is not applicable when --stage is set to converted or compiled.
--deviceId DEVICEID The serial number of the device to use. If not
available, the first in a list of queried devices will
be used for validation.
-v, --verbose Verbose printing
--host_device {x86,x86_64-windows-msvc,wos} The device that will be running conversion. Set to x86
by default.
-p ENGINE_PATH, --engine_path ENGINE_PATH
Path to the inference engine.
-w WORKING_DIR, --working_dir WORKING_DIR
Working directory for the inference_engine to store
temporary files. Creates a new directory if the
specified working directory does not exist
--output_dirname OUTPUT_DIRNAME
output directory name for the inference_engine to
store temporary files under
<working_dir>/inference_engine. Creates a new
directory if the specified working directory does not
exist
--debug_mode_off Specifies if wish to turn off debug_mode mode.
--print_version PRINT_VERSION
Print the SNPE SDK version alongside the output.
--offline_prepare Use offline prepare to run snpe model.
-bbw {8,32}, --bias_bitwidth {8,32}
option to select the bitwidth to use when quantizing
the bias. default 8
-abw {8,16}, --act_bitwidth {8,16}
option to select the bitwidth to use when quantizing
the activations. default 8
--golden_output_reference_directory GOLDEN_OUTPUT_REFERENCE_DIRECTORY, --golden_dir_for_mapping GOLDEN_DIR_FOR_MAPPING
Optional parameter to indicate the directory of the
goldens, it's used for tensor mapping without
framework.
-mn MODEL_NAME, --model_name MODEL_NAME
Name of the desired output sdk specific model.
--args_config ARGS_CONFIG
Path to a config file with arguments. This can be used
to feed arguments to the AccuracyDebugger as an
alternative to supplying them on the command line.
-wbw {8,16}, --weights_bitwidth {8,16}
option to select the bitwidth to use when quantizing
the weights. default 8
-nif, --use_native_input_files
Specifies that the input files will be parsed in the
data type native to the graph. If not specified, input
files will be parsed in floating point.
-nof, --use_native_output_files
Specifies that the output files will be generated in
the data type native to the graph. If not specified,
output files will be generated in floating point.
-qo QUANTIZATION_OVERRIDES, --quantization_overrides QUANTIZATION_OVERRIDES
Path to quantization overrides json file.
--extra_converter_args EXTRA_CONVERTER_ARGS
additional converter arguments in a quoted string.
example: --extra_converter_args 'input_dtype=data
float;input_layout=data1 NCHW'
--extra_runtime_args EXTRA_RUNTIME_ARGS
additional net runner arguments in a quoted string.
example: --extra_runtime_args
'arg1=value1;arg2=value2'
--profiling_level {off,basic,moderate,detailed,linting}
Enables profiling and sets its level.
--extra_quantizer_args EXTRA_QUANTIZER_ARGS
additional dlc quantizer arguments in a quoted string.
example: --extra_runtime_args
'arg1=value1;arg2=value2'
--fine_grain_mode FINE_GRAIN_MODE
Path to the model golden outputs required to run
inference engine using fine-grain mode.
--no_weight_quantization
Generate and add the fixed-point encoding metadata but
keep the weights in floating point
--use_symmetric_quantize_weights
Use the symmetric quantizer feature when quantizing
the weights of the model
--use_enhanced_quantizer
Use the enhanced quantizer feature when quantizing the
model
--htp_socs HTP_SOCS Specify SoC to generate HTP Offline Cache for.
--use_adjusted_weights_quantizer
Use the adjusted tf quantizer for quantizing the
weights only
--override_params Use this option to override quantization parameters
when quantization was provided from the original
source framework
Please note: All the command line arguments should either be provided through command line or through the config file. They will not override those in the config file if there is overlap.
The inference engine config file can be found in {accuracy_debugger tool root directory}/python/qti/aisw/accuracy_debugger/lib/inference_engine/configs/config_files and is a JSON file. This config file stores information that helps the inference engine know which tool and parameters to read in. Each different inference engine, and possibly engine versions in certain cases, will require its own config file.
Sample Command
snpe-accuracy-debugger \
--inference_engine \
--framework tensorflow \
--runtime cpu \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0 \
--architecture x86_64-linux-clang \
--input_list $RESOURCESPATH/samples/InceptionV3Model/data/image_list.txt \
--verbose
Sample Command
snpe-accuracy-debugger \
--inference_engine \
--framework tensorflow \
--runtime cpu \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0 \
--architecture x86_64-windows-msvc \
--input_list $RESOURCESPATH/samples/InceptionV3Model/data/image_list.txt \
--verbose
Sample Command
snpe-accuracy-debugger \
--inference_engine \
--framework tensorflow \
--runtime cpu \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0 \
--architecture wos \
--input_list $RESOURCESPATH/samples/InceptionV3Model/data/image_list.txt \
--verbose
Sample Command
snpe-accuracy-debugger \
--inference_engine \
--deviceId 357415c4 \
--framework tensorflow \
--runtime dsp \
--architecture aarch64-android \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0 \
--input_list $RESOURCESPATH/samples/InceptionV3Model/data/image_list.txt \
--verbose
Tip:
for –runtime (-r) dsp is used for snpe-net-run
the input_tensor (–i) and output_tensor (-o) does not need the :0 indexing like when runing tensorflow framework runner
two files, namely tensor_mapping.json and snpe_model_graph_struct.json are generated to be used in verification, be sure to locate these 2 files in the working_directory/inference_engine/latest
framework and golden_dir_for_mapping, or just golden_dir_for_mapping itself is an alternative to the original model to be provided to generate the tensor_mapping.json, however, providing only the golden_dir_for_mapping, the get_tensor_mapping module will try it’s best to map but it is not guaranteed the mapping would be 100% accurate.
Before running the snpe-accuracy-debugger on a Windows x86 system/Windows on Snapdragon system, ensure that you have configured the environment. And, Specify the host and target machine as x86_64-windows-msvc/wos respectively.
Note that snpe-accuracy-debugger on Windows x86 system is tested only for CPU runtime currently.
Output
Once the inference engine has finished running, it will store the output in the specified directory (or the current working directory by default) and store the files in that folder. By default, it will store the output in working_directory/inference_engine in the current working directory.
The figure above shows the sample output from one of the runs of inference engine step. The output directory contains raw files. Each raw file is an output of an op involved in various layers in the network. File input_list.txt and the directory input_list_files contain the path for sample test images. inference_engine_options.json contains all the options with which this run was launched. In addition to generating the .raw files, inference_engine also generates the model’s graph structure in a .json file. The name of the file is the same as the name of the protobuf model file. The file base.dlc is the converted model graph. The file model_graph_struct.json aids in providing the structure related information of the converted model graph during the verification step. Specifically, it helps with organizing the nodes in order (for e.g. the beginning nodes should come earlier than ending nodes). It helps to see the nodes in a streamlined manner. Finally, tensor_mapping json file contains a mapping of the various intermediate output file names generated from the framework runner step and the inference engine step.
The created .raw files are organized in the same manner as framework_runner (see above).
Verification
The Verification step compares the output (from the intermediate tensors of a given model) produced by framework diagonsis step with the one that’s produced by the inference engine step. Once the comparison is complete the verification results are compiled and displayed visually in a format that can be easily interpreted by the user.
There are different types of verifiers for e.g.: CosineSimilarity, RtolAtol, etc. To see what all verifiers are there please use the –help option like ./snpe_accuracy_debugger –verification –help. Each verifier compares the Framework Runner and Inference Engine output using an error metric. It also prepares reports and/or visualizations to help the user analyze the network’s error data.
Usage
usage: snpe-accuracy-debugger --verification [-h]
--default_verifier DEFAULT_VERIFIER
[DEFAULT_VERIFIER ...]
--golden_output_reference_directory
GOLDEN_OUTPUT_REFERENCE_DIRECTORY
--inference_results INFERENCE_RESULTS
[--tensor_mapping TENSOR_MAPPING]
[--qnn_model_json_path QNN_MODEL_JSON_PATH]
[--dlc_path DLC_PATH]
[--verifier_config VERIFIER_CONFIG]
[--graph_struct GRAPH_STRUCT] [-v]
[-w WORKING_DIR]
[--output_dirname OUTPUT_DIRNAME]
[--args_config ARGS_CONFIG]
[--target_encodings TARGET_ENCODINGS]
[-e ENGINE [ENGINE ...]]
Script to run verification.
required arguments:
--default_verifier DEFAULT_VERIFIER [DEFAULT_VERIFIER ...]
Default verifier used for verification. The options
"RtolAtol", "AdjustedRtolAtol", "TopK", "L1Error",
"CosineSimilarity", "MSE", "MAE", "SQNR", "ScaledDiff"
are supported. An optional list of hyperparameters can
be appended. For example: --default_verifier
rtolatol,rtolmargin,0.01,atolmargin,0.01 An optional
list of placeholders can be appended. For example:
--default_verifier CosineSimilarity param1 1 param2 2.
to use multiple verifiers, add additional
--default_verifier CosineSimilarity
--golden_output_reference_directory GOLDEN_OUTPUT_REFERENCE_DIRECTORY, --framework_results GOLDEN_OUTPUT_REFERENCE_DIRECTORY
Path to root directory of golden output files. Paths
may be absolute, or relative to the working directory.
--inference_results INFERENCE_RESULTS
Path to root directory generated from inference engine
diagnosis. Paths may be absolute, or relative to the
working directory.
optional arguments:
--tensor_mapping TENSOR_MAPPING
Path to the file describing the tensor name mapping
between inference and golden tensors.
--qnn_model_json_path QNN_MODEL_JSON_PATH
Path to the qnn model net json, used for transforming
axis of golden outputs w.r.t to qnn outputs. Note:
Applicable only for QNN
--dlc_path DLC_PATH Path to the dlc file, used for transforming axis of
golden outputs w.r.t to target outputs. Note:
Applicable for QAIRT/SNPE
--verifier_config VERIFIER_CONFIG
Path to the verifiers' config file
--graph_struct GRAPH_STRUCT
Path to the inference graph structure .json file. This
file aids in providing structure related information
of the converted model graph during this stage.Note:
This file is mandatory when using ScaledDiff verifier
-v, --verbose Verbose printing
-w WORKING_DIR, --working_dir WORKING_DIR
Working directory for the verification to store
temporary files. Creates a new directory if the
specified working directory does not exist
--output_dirname OUTPUT_DIRNAME
output directory name for the verification to store
temporary files under <working_dir>/verification.
Creates a new directory if the specified working
directory does not exist
--args_config ARGS_CONFIG
Path to a config file with arguments. This can be used
to feed arguments to the AccuracyDebugger as an
alternative to supplying them on the command line.
--target_encodings TARGET_ENCODINGS
Path to target encodings json file.
Arguments for generating Tensor mapping (required when --tensor_mapping is not specified):
-e ENGINE [ENGINE ...], --engine ENGINE [ENGINE ...]
Name of engine(qnn/snpe) that is used for running
inference.
Please note: All the command line arguments should either be provided through command line or through the config file. They will not override those in the config file if there is overlap.
The main verification process run using snpe-accuracy-debugger -–verification optionally uses –tensor_mapping and -–graph_struct to find files to compare. These files are generated by the inference engine step, and should be supplied to verification for best results. By default they are named tensor_mapping.json and {model name}_graph_struct.json, and can be found in the output directory of the inference engine results.
Sample Command
Compare output of framework runner with inference engine:
snpe-accuracy-debugger \
--verification \
--default_verifier CosineSimilarity param1 1 param2 2 \
--default_verifier SQNR param1 5 param2 1 \
--golden_output_reference_directory $PROJECTREPOPATH/working_directory/framework_runner/2022-10-31_17-07-58/ \
--inference_results $PROJECTREPOPATH/working_directory/inference_engine/latest/output/Result_0/ \
--tensor_mapping $PROJECTREPOPATH/working_directory/inference_engine/latest/tensor_mapping.json \
--graph_struct $PROJECTREPOPATH/working_directory/inference_engine/latest/snpe_model_graph_struct.json
Tip:
If you passed multiple images in the image_list.txt from run inference engine diagnosis, you’ll receive multiple output/Result_x, choose result that matches the input you used for framework runner for comparison (ie. in framework you used chair.raw and inference chair.raw was the first item in the image_list.txt then choose output/Result_0, if chair.raw was the second item in image_list.txt, then choose output/Result_1).
It is recommended to always supply –graph_struct and –tensor_mapping to the command as it is used to line up the report and find the corresponding files for comparison. if –tensor_mapping did not get generated from previous steps, you can supplement with –model_path, –engine, –framework to have module generate tensor_mapping during runtime.
You can also compare inference_engine outputs to inference_engine outputs by passing the /output of the inference_engine output to the –framework_results. If you want the outputs to be exact-name-matching, then you do not need to provide a tensor_mapping file.
Note that if you need to generate a tensor mapping instead of providing a path to pre-existing tensor mapping file. You can provide the –model_path option.
Verifier uses two optional config files. The first file is used to prefer parameters to specific verifiers, as well as which tensors to use these verifiers on. The second file is used to map tensor names from inference_engine to framework_runner, since certain tensors generated by framework_runner have different names than tensors generated by inference_engine.
Verifier Config:
The verifier config file is a JSON file that tells verification which verifiers (asides from the default verifier) to use and with which parameters and on what specific tensors. If no config file is provided, the tool will only use the default verifier specified from the command line, with its default parameters, on all the tensors. The JSON file is keyed by verifier names, with each verifier as its own dictionary keyed by “parameters” and “tensors”.
Config File
```json
{
"MeanIOU": {
"parameters": {
"background_classification": 1.0
},
"tensors": [["Postprocessor/BatchMultiClassNonMaxSuppression_boxes", "detection_classes:0"]]
},
"TopK": {
"parameters": {
"k": 5,
"ordered": false
},
"tensors": [["Reshape_1:0"], ["detection_classes:0"]]
}
}
```
Note that the “tensors” field is a list of lists. This is done because specific verifiers (e.g. MeanIOU) runs on two tensor at a time. Hence the two tensors are placed in a list. Otherwise if a verifier only runs on one tensor, it will have a list of lists with only one tensor name in each list.
Compare Encodings
The Compare Encodings feature is designed to compare SNPE DLC encodings with AIMET encodings. It takes a SNPE DLC file and AIMET encoding JSON files as inputs. This feature:
Extracts encodings from the given SNPE DLC file
Compares extracted SNPE encodings with given AIMET encodings
Writes results to an Excel file that highlights mismatches
Warns the user if any encodings are present in SNPE and missing from AIMET and vice-versa
Writes the extracted SNPE encodings to a JSON file for reference
Usage
usage: snpe-accuracy-debugger --compare_encodings [-h]
--input INPUT
--aimet_encodings_json AIMET_ENCODINGS_JSON
[--precision PRECISION]
[--params_only]
[--activations_only]
[--specific_node SPECIFIC_NODE]
[--working_dir WORKING_DIR]
[--output_dirname OUTPUT_DIRNAME]
[-v]
Script to compare SNPE encodings with AIMET encodings
optional arguments:
-h, --help Show this help message and exit
required arguments:
--input INPUT
Path to SNPE DLC file
--aimet_encodings_json AIMET_ENCODINGS_JSON
Path to AIMET encodings JSON file
optional arguments:
--precision PRECISION
Number of decimal places up to which comparison will be done; default is 17
--params_only Compare only parameters in the encodings
--activations_only Compare only activations in the encodings
--specific_node SPECIFIC_NODE
Display encoding differences for the given node
--working_dir WORKING_DIR
Working directory to store temporary files; the directory is created if it does not already exist
--output_dirname OUTPUT_DIRNAME
Output directory to store temporary files under <working_dir>/compare_encodings; the directory is created if it does not already exist
-v, --verbose Verbose printing
Sample Commands
# Compare both params and activations
snpe-accuracy-debugger \
--compare_encodings \
--input mv2_quantized.dlc \
--aimet_encodings_json aimet_encodings.json
# Compare only params
snpe-accuracy-debugger \
--compare_encodings \
--input mv2_quantized.dlc \
--aimet_encodings_json aimet_encodings.json \
--params_only
# Compare only activations
snpe-accuracy-debugger \
--compare_encodings \
--input mv2_quantized.dlc \
--aimet_encodings_json aimet_encodings.json \
--activations_only
# Compare only a specific encoding
snpe-accuracy-debugger \
--compare_encodings \
--input mv2_quantized.dlc \
--aimet_encodings_json aimet_encodings.json \
--specific_node /1/Conv_output_0
Tip
A working_directory is generated at the location this script is called from unless otherwise specified.
Output
The program creates a directory named latest in working_directory/compare_encodings which is symbolically linked to the most recently generated directory. In the example below, latest will have data that is symlinked to the data in the most recent directory YYYY-MM-DD_HH:mm:ss. Users may choose to override the directory name by passing it to –output_dirname, e.g., –output_dirname myTest.
The figure above shows a sample output from a compare_encodings run. The following details what each file contains.
compare_encodings_options.json contains all the options used to run this feature
encodings_diff.xlsx contains comparison results with mismatches highlighted
extracted_encodings.json contains extracted SNPE encodings
log.txt contains log statements for the run
Tensor inspection
Tensor inspection feature compares given reference output and target output tensors and dumps various statistics to represent differences between them.
The Tensor inspection feature can:
Plot histograms for golden and target tensors
Plot a graph indicating deviation between golden and target tensors
Plot a cumulative distribution graph (CDF) for golden vs target tensors
Plot a density (KDE) graph for target tensor highlighting target min/max and calibrated min/max values
Create a CSV file containing information about: target min/max; calibrated min/max; golden output min/max; target/calibrated min/max differences; and computed metrics (verifiers).
Note
Usage
usage: snpe-accuracy-debugger --tensor_inspection [-h]
--golden_data GOLDEN_DATA
--target_data TARGET_DATA
--verifier VERIFIER [VERIFIER ...]
[-w WORKING_DIR]
[--data_type {int8,uint8,int16,uint16,float32}]
[--target_encodings TARGET_ENCODINGS]
[-v]
Script to inspection tensor.
required arguments:
--golden_data GOLDEN_DATA
Path to golden/framework outputs folder. Paths may be absolute or
relative to the working directory.
--target_data TARGET_DATA
Path to target outputs folder. Paths may be absolute or relative to the
working directory.
--verifier VERIFIER [VERIFIER ...]
Verifier used for verification. The options "RtolAtol",
"AdjustedRtolAtol", "TopK", "L1Error", "CosineSimilarity", "MSE", "MAE",
"SQNR", "MeanIOU", "ScaledDiff" are supported.
An optional list of hyperparameters can be appended, for example:
--verifier rtolatol,rtolmargin,0.01,atolmargin,0,01.
To use multiple verifiers, add additional --verifier CosineSimilarity
optional arguments:
-w WORKING_DIR, --working_dir WORKING_DIR
Working directory to save results. Creates a new directory if the
specified working directory does not exist
--data_type {int8,uint8,int16,uint16,float32}
DataType of the output tensor.
--target_encodings TARGET_ENCODINGS
Path to target encodings json file.
-v, --verbose Verbose printing
Sample Commands
# Basic run
snpe-accuracy-debugger --tensor_inspection \
--golden_data golden_tensors_dir \
--target_data target_tensors_dir \
--verifier sqnr
# Pass target encodings file and enable multiple verifiers
snpe-accuracy-debugger --tensor_inspection \
--golden_data golden_tensors_dir \
--target_data target_tensors_dir \
--verifier mse \
--verifier sqnr \
--verifier rtolatol,rtolmargin,0.01,atolmargin,0.01 \
--target_encodings reference_encodings.json
Tip
A working_directory is generated from wherever this script is called from unless otherwise specified.
The figure above shows a sample output from a Tensor inspection run. The following details what each file contains.
Each tensor will have its own directory; the directory name matches the tensor name.
CDF_plots.html – Golden vs target CDF graph
Diff_plots.html – Golden and target deviation graph
Distribution_min-max.png – Density plot for target tensor highlighting target vs calibrated min/max values
Histograms.html – Golden and target histograms
golden_data.csv – Golden tensor data
target_data.csv – Target tensor data
log.txt – Log statements from the entire run
summary.csv – Target min/max, calibrated min/max, golden output min/max, target vs calibrated min/max differences, and verifier outputs
Histogram Plots
Comparison: We compare histograms for both the golden data and the target data.
Overlay: To enhance clarity, we overlay the histograms bin by bin.
Binned Ranges: Each bin represents a value range, showing the frequency of occurrence.
Visual Insight: Overlapping histograms reveal differences or similarities between the datasets.
Interactive: Hover over histograms to get tensor range and frequencies for the dataset.
Cumulative Distribution Function (CDF) Plots
Overview: CDF plots display the cumulative probability distribution.
Overlay: We superimpose CDF plots for golden and target data.
Percentiles: These plots illustrate data distribution across different percentiles.
Hover Details: Exact cumulative probabilities are available on hover.
Tensor Difference Plots
Inspection: We generate plots highlighting differences between golden and target data tensors.
Scatter and Line: Scatter plots represent tensor values, while line plots show differences at each index.
Interactive: Hover over points to access precise values.
Tensor Mapping:
Tensor mapping is a JSON file keyed by inference tensor names, of framework tensor names. If the tensor mapping is not provided, the tool will assume inference and golden tensor names are identical.
Tensor Mapping File
```json
{
"Postprocessor/BatchMultiClassNonMaxSuppression_boxes": "detection_boxes:0",
"Postprocessor/BatchMultiClassNonMaxSuppression_scores": "detection_scores:0"
}
```
Output
Verification’s output is divided into different verifiers. For example, if both RtolAtol and TopK verifiers are used, there will be two sub-folders named “RtolAtol” and “TopK”. Availble verifiers can be found by just issuing –help option.
Under each sub-folder, the verification analysis for each tensor is organized similar to how framework_runner and inference_engine (see above) are organized. For each tensor, a CSV and HTML file is generated. In addition to the tensor-specific analysis, the tool also generates a summary CSV and HTML file which summarizes the data from all verifiers and their subsequent tensors. The following figure shows how a sample summary generated in the verification step looks. Each row in this summary corresponds to one tensor name that is identified by the framework runner and inference engine steps. The final column shows cosinesimilarity score which can vary between 0 to 1 (this range might be different for other verifiers). If the score is high enough then it means that the result produced by both the steps for that particular tensor are fairly similar. However, if the score is too low then it gives the developer a point of inspection. The developer can then further investigate those specific tensors (if multiple) into details. Developer should inspect tensors from top-to-bottom order, meaning if a tensor is broken at an earlier node, anything that was generated post that node is unreliable until that node is properly fixed. Hence, this process might help in improving the overall accuracy of the sdk. That is how this tool helps in debugging.
Run SNPE Accuracy Debugger E2E:
This feature is designed to run the framework runner, inference engine, and verification features sequentially with a single command to debug the model. It provides quick analysis to identify layers of model causing accuracy deviation. Additionally, tensor inspection (when –enable_tensor_inspection is passed) is executed which dumps various plots for all intermediate outputs.
Usage
usage: snpe-accuracy-debugger [--framework_runner] [--inference_engine] [--verification] [-h]
Script that runs Framework Runner, Inference Engine or Verification.
Arguments to select which component of the tool to run. Arguments are mutually exclusive (at most 1 can be selected). If none are selected, then all components are run:
--framework_runner Run framework
--inference_engine Run inference engine
--verification Run verification
optional arguments:
-h, --help Show this help message. To show help for any of the
components, run script with --help and --<component>.
For example, to show the help for Framework Runner,
run script with the following: --help
--framework_runner
usage: snpe-accuracy-debugger [-h] -f FRAMEWORK [FRAMEWORK ...] -m MODEL_PATH
-i INPUT_TENSOR [INPUT_TENSOR ...] -o
OUTPUT_TENSOR -r RUNTIME -a
{aarch64-android,x86_64-linux-clang}
-l INPUT_LIST --default_verifier DEFAULT_VERIFIER
[DEFAULT_VERIFIER ...] [-v] [-w WORKING_DIR]
[--output_dirname OUTPUT_DIRNAME]
[--deep_analyzer {modelDissectionAnalyzer}]
Options for running the Accuracy Debugger components
optional arguments:
-h, --help show this help message and exit
Arguments required by both Framework Runner and Inference Engine:
-f FRAMEWORK [FRAMEWORK ...], --framework FRAMEWORK [FRAMEWORK ...]
Framework type and version, version is optional.
-m MODEL_PATH, --model_path MODEL_PATH
Path to the model file(s).
-i INPUT_TENSOR [INPUT_TENSOR ...], --input_tensor INPUT_TENSOR [INPUT_TENSOR ...]
The name, dimensions, raw data, and optionally data
type of the network input tensor(s) specifiedin the
format "input_name" comma-separated-dimensions path-
to-raw-file, for example: "data" 1,224,224,3 data.raw
float32. Note that the quotes should always be
included in order to handle special characters,
spaces, etc. For multiple inputs specify multiple
--input_tensor on the command line like:
--input_tensor "data1" 1,224,224,3 data1.raw
--input_tensor "data2" 1,50,100,3 data2.raw float32.
-o OUTPUT_TENSOR, --output_tensor OUTPUT_TENSOR
Name of the graph's specified output tensor(s).
Arguments required by Inference Engine:
-r RUNTIME, --runtime RUNTIME
Runtime to be used for inference.
-a {aarch64-android,x86_64-linux-clang}, --architecture {aarch64-android,x86_64-linux-clang}
Name of the architecture to use for inference engine.
-l INPUT_LIST, --input_list INPUT_LIST
Path to the input list text.
Arguments required by Verification:
--default_verifier DEFAULT_VERIFIER [DEFAULT_VERIFIER ...]
Default verifier used for verification. The options
"RtolAtol", "AdjustedRtolAtol", "TopK", "L1Error",
"CosineSimilarity", "MSE", "MAE", "SQNR", "MeanIOU",
"ScaledDiff" are supported. An optional list of
hyperparameters can be appended. For example:
--default_verifier
rtolatol,rtolmargin,0.01,atolmargin,0,01. An optional
list of placeholders can be appended. For example:
--default_verifier CosineSimilarity param1 1 param2 2.
to use multiple verifiers, add additional
--default_verifier CosineSimilarity
optional arguments:
-v, --verbose Verbose printing
-w WORKING_DIR, --working_dir WORKING_DIR
Working directory for the wrapper to store temporary
files. Creates a new directory if the specified
working directory does not exitst.
--output_dirname OUTPUT_DIRNAME
output directory name for the wrapper to store
temporary files under <working_dir>/wrapper. Creates a
new directory if the specified working directory does
not exist
--golden_output_reference_directory
Optional parameter to indicate the directory of the golden reference outputs.
When this option is provided, the framework runner is step skipped.
In inference engine step, it's used for tensor mapping without a framework.
In verification step, it's used as a reference to compare
outputs produced in the inference engine step.
--enable_tensor_inspection
Plots graphs (line, scatter, CDF etc.) for each
layer's output. Additionally, summary sheet will have
more details like golden min/max, target min/max etc.,
--deep_analyzer {modelDissectionAnalyzer}
Deep Analyzer to perform deep analysis
Sample Command
snpe-accuracy-debugger \
--framework tensorflow \
--runtime cpu \
--model_path $RESOURCESPATH/samples/InceptionV3Model/inception_v3_2016_08_28_frozen.pb \
--input_tensor "input:0" 1,299,299,3 $RESOURCESPATH/samples/InceptionV3Model/data/chairs.raw \
--output_tensor InceptionV3/Predictions/Reshape_1:0 \
--architecture x86_64-linux-clang \
--input_list $RESOURCESPATH/samples/InceptionV3Model/data/image_list.txt \
--default_verifier CosineSimilarity \
--enable_tensor_inspection \
--verbose
Note
The –enable_tensor_inspection argument significantly increases overall execution time when used with large models. To speed up execution, omit this argument.
Output
The program creates framework_runner, inference_engine, verification, and wrapper output directories as below:
framework_runner – Contains a timestamped directory that contains the intermediate layer outputs (framework) stored in .raw format as described in the framework runner step.
inference_engine – Contains a timestamped directory that contains the intermediate layer outputs (inference engine) stored in .raw format as described in the inference engine step.
verification directory – Contains a timestamped directory that contains the following:
A directory for each verifier specified while running oneshot; it contains CSV and HTML files with metric details for each layer output
tensor_inspection – Individual directories for each layer’s output with the following contents:
CDF_plots.png – Golden vs target CDF graph
Diff_plots.png – Golden and target deviation graph
Histograms.png – Golden and target histograms
golden_data.csv – Golden tensor data
target_data.csv – Target tensor data
summary.csv – Report for verification results of each layers output
Wrapper directory containing log.txt with the entire log for the run.
Note: Except wrapper directory all other directories will have a folder called latest which is a symlink to the latest run’s corresponding timestamped directory.
Snapshot of summary.csv file:
Understanding the oneshot-layerwise report:
Column |
Description |
|---|---|
Name |
Output name of the current layer |
Layer Type |
Type of the current layer |
Size |
Size of this layer’s output |
Tensor_dims |
Shape of this layer’s output |
<Verifier name> |
Verifier value of the current layer output compared to reference output |
golden_min |
minimum value in the reference output for current layer |
golden_max |
maximum value in the reference output for current layer |
target_min |
minimum value in the target output for current layer |
target_max |
maximum value in the target output for current layer |
Quantization Checker
The quantization checker analyzes activations, weights, and biases of a given model. It provides:
Comparision between quantized and unquantized weights and biases
Analysis on unquantized weights, biases, and activations
Results in csv, html, or plots
Problematic weights and biases for a given bitwidth quantization
Usage
usage: qnn-accuracy-debugger --quant_checker [-h] \
--model_path \
--input_tensor \
--config_file \
--framework \
--input_list \
--output_tensor \
[--engine_path] \
[--working_dir] \
[--quantization_overrides] \
[--extra_converter_args] \
[--bias_width] \
[--weights_width] \
[--host_device] \
[--deviceId] \
[--generate_csv] \
[--generate_plots] \
[--per_channel_plots] \
[--golden_output_reference_directory] \
[--output_dirname]
[--verbose]
Sample quant_checker_config_file
- {
- “WEIGHT_COMPARISON_ALGORITHMS”: [
{“algo_name”:”minmax”,”threshold”:”10”}, {“algo_name”:”maxdiff”, “threshold”:”10”}, {“algo_name”:”sqnr”, “threshold”:”26”}, {“algo_name”:”stats”, “threshold”:”2”}, {“algo_name”:”data_range_analyzer”}, {“algo_name”:”data_distribution_analyzer”, “threshold”:”0.6”}
], “BIAS_COMPARISON_ALGORITHMS”: [
{“algo_name”:”minmax”, “threshold”:”10”}, {“algo_name”:”maxdiff”, “threshold”:”10”}, {“algo_name”:”sqnr”, “threshold”:”26”}, {“algo_name”:”stats”, “threshold”:”2”}, {“algo_name”:”data_range_analyzer”}, {“algo_name”:”data_distribution_analyzer”, “threshold”:”0.6”}
], “ACT_COMPARISON_ALGORITHMS”: [
{“algo_name”:”minmax”, “threshold”:”10”}, {“algo_name”:”data_range_analyzer”}
], “INPUT_DATA_ANALYSIS_ALGORITHMS”: [{“algo_name”:”stats”, “threshold”:”2”}], “QUANTIZATION_ALGORITHMS”: [“cle”, “None”], “QUANTIZATION_VARIATIONS”: [“tf”, “enhanced”, “symmetric”, “asymmetric”]
}
Output
Output are available in the <working-directory>/results, which looks like: .. container:
.. figure:: /../images/quant_checker_acc_debug_output_dir_struct.png
Results are provided in:
HTML
CSV
Histogram
A log is provided in the <working-directory>/quant_checker directory.
HTML Each HTML file contains a summary of the results for each quantization option and for each input file provided.
The following example provides additional guidance on the contents of the HTML files.
CSV
Each CSV file contains detailed computation results for a specific node type (activation/weight/bias) and quantization option. Each row in the csv file displays the op name, node name, passes accuracy (True/False), computation result (accuracy differences), threshold used for each algorithm, and the algorithm name. The format of the computation results (accuracy differences) differs according to the algorithms/metrics used.
The following table provides additional notes about the different algorithms and information in each csv row.
The following CSV example shows weight data for one of the quantization options.
Separate .csv files are generated for activations, weights, and biases for each quantization option. The activation-related results include analysis for each input file provided.
Histogram
For each quantization variation and for each weight and bias tensor in the model, we generate historagm. a histogram is generated for each quantization variation and for each weight and bias tensor in the model. The following example illustrates the generated histograms.
Logs
The log files contain the following information.
The commands executed as part of the script’s run, including different runs of the snpe-converter tool with different quantization options
Analysis failures for activations, weights, and biases
The following example shows a sample log output.
<====ACTIVATIONS ANALYSIS FAILURES====>
<====ACTIVATIONS ANALYSIS FAILURES====>
Results for the enhanced quantization: | Op Name | Activation Node | Passes Accuracy | Accuracy Difference | Threshold Used | Algorithm Used | | conv_tanh_comp1_conv0 | ReLU_6919 | False | minabs_diff: 0.59 maxabs_diff: 17.16 | 0.05 | minmax |
where,
Op Name : Op name as expressed in snpe_model.dlc
Activation Node : Activation node name in the operation
Passes Accuracy : True if the quantized activation (or weight or bias) meets threshold when compared with values from float32 graph; false otherwise
Accuracy Difference : Details about the accuracy per the algorithm used
Threshold Used : The threshold used to influence the result of “Passes Accuracy” column
Algorithm Used : Metric used to compare actual quantized activations/weights/biases against unquantized float data or analyze the quality of unquantized float data. Metrics can be minmax, maxdiff, sqnr, stats, data_range_analyzer, data_distribution_analyzer.