#DoRA: Weight-Decomposed Low-Rank Adaptation

The efficiency of fine-tuning large language models has long been governed by the trade-off between parameter count and expressive power. Standard Low-Rank Adaptation (LoRA) reduced the computational barrier by confining weight updates to a low-dimensional space, yet a performance gap persisted between these sparse updates and full parameter fine-tuning. Research into the behavioral patterns of these methods revealed that LoRA updates are limited by a rigid coupling between magnitude and direction. In LoRA, any significant change in the orientation of a weight vector is almost always accompanied by a proportional increase in its magnitude, a constraint that does not exist in full-parameter optimization. This lack of flexibility prevents the model from executing nuanced adjustments, such as precise directional shifts that require minimal changes in scale.

#Weight Decomposition and Decoupling

Magnitude and direction updates of (a) FT, (b) LoRA, and (c) DoRA across different layers.

The mechanism of Weight-Decomposed Low-Rank Adaptation (DoRA) addresses this coupling by reparameterizing the pre-trained weight matrix $W$ into two distinct components: a magnitude vector $m$ and a directional matrix $V$. By applying the standard LoRA update specifically to the directional component while allowing the magnitude to be learned independently, the model can navigate the optimization landscape with a level of granularity previously reserved for full fine-tuning. This mathematical decoupling allows the gradient to be conditioned more effectively, as directional shifts no longer force an unintended growth in weight magnitude. This finding revealed that the bottleneck in sparse adaptation was not the number of parameters being updated, but the structural constraints on how those parameters were allowed to move through the function space.