Hexagon NPU Runtime Driver (Windows Only)

Hexagon NPU Runtime Driver (HNRD) is available for Windows based Snapdragon® X Series Platforms. HNRD is designed to be forward and backward compatible with Qualcomm AI Stack SDKs. With HNRD in the system, applications have the option to unbundle from SNPE HTP platform dependent libraries, and allows these applications to be portable over to older and newer Windows platforms. HNRD is packaged and distributed independently from the SNPE SDK and it is currently packaged with the device BSP. OEMs use the BSP to pre-install HNRD on their devices.

Switching Between Traditional and HNRD Paths

Pre-driver:

• Traditional path only; applications build with the SNPE SDK and bundle the SNPE HTP platform dependent libraries

Post-driver:

• Traditional and HNRD paths available; the application can choose which path to use

• Traditional path; applications build with the SNPE SDK and bundle the SNPE HTP platform dependent libraries with the application (Default – same as pre-driver)

• HNRD path; applications build with the SNPE SDK, but utilize the platform dependent libraries installed on the system

• Note: there is no difference in the build step between traditional and HNRD paths. In addition, the bundling and choosing between traditional and HNRD paths is applicable to both QNN and SNPE

In other words, if an application bundles the SNPE HTP platform dependent libraries (i.e., SnpeHtpV73Stub.dll and SnpeHtpV73Skel.so), it will default to choose the traditional path. Otherwise, if the platform dependent libraries are not bundled, it will fall back to choose the HNRD path.

The following sample logs illustrate the HNRD path being applied:

  0.0ms [WARNING] QnnDsp <W> Traditional path not available. Switching to user driver path
  0.0ms [WARNING] QnnDsp <W> HTP user driver is loaded. Switched to user driver path

Compatibility Support

The minimum QNN and SNPE version is 2.22.2. Applications can be built with older or newer versions of the SNPE SDK and they will still work on the device. Depending on the HNRD version installed on the system, new features may not be supported.

Note

Traditional and HNRD paths can co-exist on the platform and each application independently selects whether to use traditional or HNRD paths.

Model Cache Management

When utilizing Init Caching with HNRD, it requires special attention on managing the caches. When using HNRD path, online prepare and cache loading are done by HNRD since they are platform-dependent. A cache generated by one version of HNRD might not be able to run on HNRD with an older version or might not utilize all software / hardware capabilities on HNRD with a new version. It is required to check the compatibility of saved caches with the HNRD every time before loading, since the HNRD installed on a device can be upgraded (or downgraded) at any time.

Snpe_SNPEBuilder_Build() and SNPEBuilder::build() check the compatibility automatically before loading the caches. Calling Snpe_SNPEBuilder_SetCacheCompatiblityMode() and SNPEBuilder::setCacheCompatibilityMode() can control whether to fail sub-optimal caches during compatibility check. Snpe_SNPEBuilder_ValidateCache() and SNPEBuilder::validateCache() do a similar check as Snpe_SNPEBuilder_Build() and SNPEBuilder::build() except they won’t create SNPE instances.

If a cache fails to pass the compatibility check, performing online prepare to create another valid cache can solve the problem. To reduce the latency impact of online prepare, continue execution with the original sub-optimal cache while doing online prepare in a background thread and switch to the new cache once online prepare is done.

Note that online prepare may not succeed. This can happen when a model converted by a newer SDK version uses features that are not supported by the HNRD. In such a case, upgrade the HNRD. The following example shows how to handle cache management.

Snpe_SNPE_Handle_t snpeHandle;  // SNPE used to do inference
std::future<Snpe_SNPEBuilder_Handle_t>
    futureSnpeHandle;  // SNPE being created in background

Snpe_DlContainer_Handle_t containerHandle =
    Snpe_DlContainer_Open(dlcPath.string().c_str());
Snpe_RuntimeList_Handle_t runtimeListHandle = Snpe_RuntimeList_Create();
Snpe_SNPEBuilder_Handle_t snpeBuilderHandle = Snpe_SNPEBuilder_Create(containerHandle);

// Add DSP runtime.
Snpe_RuntimeList_Add(runtimeListHandle, SNPE_RUNTIME_DSP);
Snpe_SNPEBuilder_SetRuntimeProcessorOrder(snpeBuilderHandle, runtimeListHandle);

// Set compatbility mode to strict to check sub-optimality.
Snpe_SNPEBuilder_SetCacheCompatibilityMode(snpeBuilderHandle,
                                           SNPE_CACHE_COMPATIBILITY_STRICT);

// Create SNPE.
snpeHandle = Snpe_SNPEBuilder_Build(snpeBuilderHandle);

if (snpeHandle == NULL) {
  Snpe_ErrorCode_t error = Snpe_ErrorCode_getLastErrorCode();
  if (error == SNPE_ERRORCODE_DLCACHING_SUBOPTIMAL_CACHE) {
    // The cache is valid but sub-optimal.
    // Continue exection with this cache.
    // Set compatiblity mode to permissive to bypass sub-optimality check.
    Snpe_SNPEBuilder_SetCacheCompatibilityMode(snpeBuilderHandle,
                                               SNPE_CACHE_COMPATIBILITY_PERMISSIVE);
    snpeHandle = Snpe_SNPEBuilder_Build(snpeBuilderHandle);

    // Create another SNPE with better performance in background thread.
    // Set compatibility mode to generate new cache.
    Snpe_SNPEBuilder_SetCacheCompatibilityMode(
        snpeBuilderHandle, SNPE_CACHE_COMPATIBILITY_ALWAYS_GENERATE_NEW_CACHE);
    // Enable init cache mode to save the cache later.
    Snpe_SNPEBuilder_SetInitCacheMode(snpeBuilderHandle, 1);
    futureSnpeHandle =
        std::async(std::launch::async, Snpe_SNPEBuilder_Build, snpeBuilderHandle);
  } else if (error == SNPE_ERRORCODE_QNN_CONTEXT_ERROR_CREATE_FROM_BINARY ||
             error == SNPE_ERRORCODE_QNN_COMMON_ERROR_NOT_SUPPORTED) {
    // The cache cannot run.
    // Force generate a new cache.
    Snpe_SNPEBuilder_SetCacheCompatibilityMode(
        snpeBuilderHandle, SNPE_CACHE_COMPATIBILITY_ALWAYS_GENERATE_NEW_CACHE);
    snpeHandle = Snpe_SNPEBuilder_Build(snpeBuilderHandle);

    if (snpeHandle == NULL) {
      // If it still fails, one possible reason could be HNRD is too old for the
      // graph. In such case, prompt the users to upgrade HNRD.
      cleanUp(containerHandle, runtimeListHandle, snpeBuilderHandle);
      message(ERROR, "The HNRD is too old. Please install latest HNRD.");
    }
  } else {
    cleanUp(containerHandle, runtimeListHandle, snpeBuilderHandle);
    throw std::runtime_error("Skip handling of other errors.");
  }
}

// Do inference.
while (waitInputData()) {
  // Switch to new SNPE if prepare is done.
  if (futureSnpeHandle.valid() &&
      futureSnpeHandle.wait_for(std::chrono::seconds(0)) == std::future_status::ready) {
    // Clean up the original SNPE.
    Snpe_SNPE_Delete(snpeHandle);
    // Switch to the new SNPE.
    snpeHandle = futureSnpeHandle.get();
    // Save the new cache to DLC.
    Snpe_DlContainer_Save(containerHandle, dlcPath.string().c_str());
  }

  doInference(snpeHandle);
}

// Clean up.
Snpe_SNPE_Delete(snpeHandle);
cleanUp(containerHandle, runtimeListHandle, snpeBuilderHandle);