MoreLoRA

MoreLoRA

Original LoRA:

$W = W_0 + UV$ and $rank(UV)leq r$

Better Initialization:

$W = W_0 - U_0{V_0} + UV$

Additive LoRA:

$W = W_0 + UI_{r(1times frac{n}{r})}+I_{r(frac{m}{r}times 1)}V$ where $Uin mathbb{R}^{mtimes r}, Vin{mathbb{R}^{r times n}}$ and $rank(UV)leq 2r$

Hadamard Mul LoRA:

$W = W_0 + odot_{i=1}^{i=k}(Delta_i)$ where $Delta_i = U_iV_i$

$r'= frac{r}{k},U_iinmathbb{R}^{mtimes r'}$, $V_iinmathbb{R}^{r'times n}$ and $rank(odot_{i=1}^{i=k}(Delta_i^T))leq (frac{r}{k})^k$

Hadamard Add LoRA:

$W = W_0 + odot_{i=1}^{i=k}(Delta_i)$ where $Delta_i = U_iI_{r'(1times frac{n}{r'})}+I_{r'(frac{m}{r'}times 1)}V_i$

$r'= frac{r}{k}$, $U_iinmathbb{R}^{r'times n}$, $V_iinmathbb{R}^{mtimes r'}$ and $rank(odot_{i=1}^{i=k}(Delta_i))leq (frac{2r}{k})^k$

Hadamard LoRA: Activation

$Delta = odot_{i=1}^{i=k}(tanh(U_iV_i^T))$

$Delta = odot_{i=1}^{i=k}(sigma(U_iV_i^T)) $

DyLoRA:

randomly update a series of ranks

Update part of the layers

Reference:

@online{kexuefm-9590,
    title={梯度视角下的LoRA：简介、分析、猜测及推广},
    author={苏剑林},
    year={2023},
    month={Apr},
    url={url{https://spaces.ac.cn/archives/9590}},
}

@misc{hyeonwoo2023fedpara,
      title={FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning}, 
      author={Nam Hyeon-Woo and Moon Ye-Bin and Tae-Hyun Oh},
      year={2023},
      eprint={2108.06098},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}