Basic Dialog - LoRA

See Genie Dialog JSON configuration string for details on the fields and what they mean. An example model_config can be found at ${QNN_SDK_ROOT}/examples/Genie/configs/llama2-7b-htp-lora.json. Note that the tokenizer path and context binary fields will need to be updated based on your actual preparation steps.

Note

Use LLaMA-2-7b notebook’s for generating AR-N models.

To run using LoRA on QNN HTP backend, open a command shell on android and run the following. This assumes that the HTP architecture has been set (e.g., ARCH=79). Please use the steps mentioned above for libraries, binaries, tokenizer and backend_ext_config.

adb shell mkdir -p /data/local/tmp/
adb push <path to llama2-7b-htp-lora.json> /data/local/tmp/
adb push <path to lora bin files> /data/local/tmp/

# open adb shell
adb shell
export LD_LIBRARY_PATH=/data/local/tmp/
export PATH=$LD_LIBRARY_PATH:$PATH

cd $LD_LIBRARY_PATH
./genie-t2t-run -c <path to llama2-7b-htp-lora.json>
                -p "What is the most popular cookie in the world?"
                -l lora1,alpha,0.5