LoRA (Low-Rank Adaptation)

LoRA is primarily focused on modifying large pre-trained models with minimal additional parameters, thereby making the adaptation process more efficient and less resource-intensive. It works by introducing trainable low-rank matrices that adapt the self-attention and feed-forward layers of a transformer model. These matrices adjust the pre-trained weights indirectly rather than modifying them directly, allowing significant changes in the model's behavior with only a small increase in the number of parameters. This method is particularly useful in scenarios where deploying fully fine-tuned models is computationally expensive or impractical.

The concept of Low-Rank Adaptation emerged around 2021, developed by researchers at Microsoft. It gained attention for its efficacy in adapting large models like GPT-3 and BERT with minimal computational overhead.

The development of LoRA was led by Edward J. Hu and other collaborators at Microsoft Research. Their work demonstrates a significant step forward in making large model tuning more accessible and efficient, particularly for applications where computational resources are a limiting factor.

LoRA
Low-Rank Adaptation

Key Contributors

Newsletter

Academic Papers

Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation

Sparse low-rank adaptation of pre-trained language models

The expressive power of low-rank adaptation

Lora+: Efficient low rank adaptation of large models

Modify mistral large performance with low-rank adaptation (lora) on the big-bench dataset

LoRALow-Rank Adaptation