LoRA Introduction

• LoRA = Low Rank Adaptation

• LoRA v1 : LoRA weights as inputs

  • LoRA supported by transforming the model and applying LoRA weights as inputs

• LoRA v2 : Native LoRA

  • LoRA weights are applied “natively” into the graph as weights at attachment points

  • Advantages : performance, regained accuracy on Quantization

• LoRA Terminology :

  • Adapter : set of LoRA weights for one use case

  • Alpha : parameter for setting the strength