SPD Dialog - Android

See Genie Dialog JSON configuration string for details on the fields and what they mean. An example model_config can be found at ${QNN_SDK_ROOT}/examples/Genie/configs/llama2-7b-draft-htp-target-htp-spd.json. Note that the tokenizer path and context binary fields will need to be updated based on your actual preparation steps.

Note

Use LLaMA-2-7b notebook’s for generating AR-N models for target.

Note

Use Small Sized LLM models ex. 115M for draft.

To run using SSD on QNN HTP backend, open a command shell on android and run the following. This assumes that the HTP architecture has been set (e.g., ARCH=79). Please use the steps mentioned above for libraries, binaries, tokenizer and backend_ext_config. .. note:: If using different backends for target and draft, ensure the libraries and binaries both the engines are present.

adb shell mkdir -p /data/local/tmp/
adb push <path to llama2-7b-draft-htp-target-htp-spd.json> /data/local/tmp/

# open adb shell
adb shell
export LD_LIBRARY_PATH=/data/local/tmp/
export PATH=$LD_LIBRARY_PATH:$PATH

cd $LD_LIBRARY_PATH
./genie-t2t-run -c <path to llama2-7b-draft-htp-target-htp-spd.json>
                -p "What is the most popular cookie in the world?"