GenieDialog_tokenQuery

See Genie Dialog JSON configuration string for details on the fields and what they mean. An example model_config can be found at ${QNN_SDK_ROOT}/examples/Genie/configs/llama2-7b-genaitransformer.json.

Note

Use LLaMA-2-7b notebook’s for generating AR-N models.

adb shell mkdir -p /data/local/tmp/
adb push <path to llama2-7b-genaitransformer.json> /data/local/tmp/
adb push <path to token file(.txt)> /data/local/tmp/

# open adb shell
adb shell
export LD_LIBRARY_PATH=/data/local/tmp/
export PATH=$LD_LIBRARY_PATH:$PATH
export ADSP_LIBRARY_PATH=$LD_LIBRARY_PATH

cd $LD_LIBRARY_PATH
./genie-t2t-run -c <path to llama2-7b-genaitransformer-htp-kv-share.json>
                -tok <path to token file(.txt)>

# Example tokenfile.txt
24948 592 1048 15146 2055

Model inference using token to token feature on Windows

Open Developer PowerShell for VS2022 on Windows on Snapdragon host and run:

# Make sure environment is setup as per instructions, or can cd into bin folder on Windows host
cd <QNN_SDK_ROOT>\bin\aarch64-windows-msvc
.\genie-t2t-run.exe -c <path to cpu-model-config.json>
                    -tok <path to token file(.txt)>

# Example tokenfile.txt
24948 592 1048 15146 2055