Custom Operator Support

Qualcomm® AI Engine Direct Delegate supports custom TFLite operators through Qualcomm® AI Engine Direct Op Package mechanism. Custom TFLite operators, with kTfLiteBuiltinCustom TfLiteBuiltinOperator type can be successfully delegated and supported through user written op packages. Note that builtin TFLite nodes are not currently supported with op packages.

Note that Qualcomm® AI Engine Direct SDK is necessary to compile an OP package.

Op Package Format

In order for the delegate to support the parameters of custom TFLite operators, it requires a specific op package format to be used. Specifically it should have the following format:

Inputs

in[0]…in[m-1]

The same number of input tensors as defined in the TFLite graph. Where m is the number of inputs.

  • Mandatory: true

  • Data type: backend specific

  • Shape: Any

Parameters

CustomInitialData

A binary blob representing the TfLiteNode::custom_initial_data, if it is not empty in the model. The delegate views this blob as opaque and simply passes it down. It is the responsibility of the op package to properly parse this blob.

  • Mandatory: true, if you OP have custom_data. Or, just don’t create this parameter.

  • Data type: QNN_DATATYPE_UINT_8

  • Shape: [b], b = size in bytes of the blob

Outputs

out[0]…out[n-1]

The same number of output tensors as defined in the TFLite graph. Where n is the number of outputs.

  • Mandatory: true

  • Data type: backend specific

  • Shape: Any

Consult the Qualcomm® AI Engine Direct SDK documentation for information on generating op packages.

Registering and Running Op Packages

After an op package library has been generated, certain information needs to be passed to the delegate in order to properly delegate the nodes. The following code example shows how to register an op package with the delegate’s C interface.

// Define the mapping between TFLite operator and op type in op package
TfLiteQnnDelegateOpPackageOpMap ops_map;

// Assumed the TFLite custom OP name is ExampleCustomOp
ops_map.custom_op_name = "ExampleCustomOp";
ops_map.qnn_op_type_name = "HtpExampleOp";

// Set the op package info. Note that we need two package info for HTP.
TfLiteQnnDelegateOpPackageInfo op_package_info[2];

// The first package info is for CPU, which is used to prepare HTP graph
op_package_info[0].op_package_name = "HtpExampleOpPackage";
op_package_info[0].op_package_path = "HtpExampleOpPackage/build/aarch64-android/libQnnHtpExampleOpPackage.so";
op_package_info[0].interface_provider = "HtpExampleOpPackageInterfaceProvider";
op_package_info[0].target = "CPU";
op_package_info[0].num_ops_map = 1;
op_package_info[0].ops_map = &ops_map;

// The second one is on HTP, for graph execution.
op_package_info[1].op_package_name = "HtpExampleOpPackage";
op_package_info[1].op_package_path = "HtpExampleOpPackage/build/hexagon-v68/libQnnHtpExampleOpPackage.so";
op_package_info[1].interface_provider = "HtpExampleOpPackageInterfaceProvider";
op_package_info[1].target = "HTP";
op_package_info[1].num_ops_map = 1;
op_package_info[1].ops_map = &ops_map;

// Set the op package info into the options structure
TfLiteQnnDelegateOptions options = TfLiteQnnDelegateOptionsDefault();
options.op_package_options.num_op_package_infos = 2;
options.op_package_options.op_package_infos = op_package_info;

Note that providing two different op packages that support the same TFLite custom operator is prohibited. However, there is an exception when the different op packages have the same package name. For example, it is valid to provide a “CPU” target of an op package along with a “HTP” variant that both support the custom operator, as long as both have the same package name.

When checking whether the node is supported by an op package, the version of the node is not checked. It is up to the user to determine whether the op package supports this version of the custom operator.