QNN Gen AI Transformer

Model download

Download bge-large-en-v1.5 from https://huggingface.co/BAAI/bge-large-en-v1.5

Model conversion

The following section demonstrates converting a model using qnn-genai-transformer-composer.

Model conversion on Linux and Android

Open a command shell on Linux host and run:

# Make sure environment is setup as per instructions, or can cd into bin folder on Linux host
cd ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/
./qnn-genai-transformer-composer --outfile <output filename with complete path>.bin
                                 --model <path-to-downloaded-BGE-model-directory>

Model configuration

See Genie Embedding JSON configuration string for details on the fields and what they mean. An example model config can be found at ${QNN_SDK_ROOT}/examples/Genie/configs/bge-large-genaitransformer.json. Note that the tokenizer path and model bin fields will need to be updated based on your actual preparation steps.

Model execution

The following section demonstrates running a model on the QNN GenAITransformer backend using genie-t2e-run.

Note

Results will be saved in output.raw file in the working directory.

Model execution on Linux

Open a command shell on Linux host and run:

# Make sure environment is setup as per instructions, or can cd into bin folder on Linux host
cd <QNN_SDK_ROOT>\bin\x86_64-linux-clang
./genie-t2e-run -c <path to bge-large-genaitransformer.json>
                -p "Tell me about Qualcomm"

Model execution on Android

Open a command shell on Linux host and run:

# make sure a test device is connected
adb devices

# push artifacts to device
adb push ${QNN_SDK_ROOT}/bin/aarch64-android/genie-t2e-run /data/local/tmp/
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libGenie.so /data/local/tmp/
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnGenAiTransformer.so /data/local/tmp/
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnGenAiTransformerCpuOpPkg.so /data/local/tmp/
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnGenAiTransformerModel.so /data/local/tmp/
adb push <path to tokenizer.json> /data/local/tmp/
adb push <path to bge-large-genaitransformer.json> /data/local/tmp/
adb push <path to model bin file, e.g. <path-to-downloaded-BGE-model-directory>/model.bin> /data/local/tmp/

# open adb shell
adb shell

export LD_LIBRARY_PATH=/data/local/tmp/
export PATH=$LD_LIBRARY_PATH:$PATH

cd $LD_LIBRARY_PATH
./genie-t2e-run -c <path to bge-large-genaitransformer.json>
                -p "Tell me about Qualcomm"