MoreLoRA

Morelare

Lora original:

$ W = w_0 + uv $ et $ RANK (UV) leq r $

Meilleure initialisation:

$ W = w_0 - u_0 {v_0} + uv $

Additive Lora:

$ W = w_0 + ui_ {r (1 Times frac {n} {r})} + i_ {r ( frac {m} {r} Times 1)} v $ où $ U in mathbb {r} ^ {m Times R}, v in { mathbb {r} ^ {r Times N}} $ et $ RANK (UV) leq 2r $

Hadamard Mul Lora:

$ W = w_0 + odot_ {i = 1} ^ {i = k} ( delta_i) $ où $ Delta_i = u_iv_i $

$ r '= frac {r} {k}, u_i in mathbb {r} ^ {M Times R'} $ , $ V_i in mathbb {r} ^ {r ' Times n} $ et $ rank ( odot_ {i = 1} ^ {i = k} ( delta_i ^ t)) leq ( frac {r} {k}) ^ k $

Hadamard Ajouter Lora:

$ W = w_0 + odot_ {i = 1} ^ {i = k} ( delta_i) $ où Dollars

$ r '= frac {r} {k} $ , $ U_i in mathbb {r} ^ {r ' Times n} $ , $ V_i in mathbb {r} ^ {M Times R '} $ et $ rank ( odot_ {i = 1} ^ {i = k} ( delta_i)) leq ( frac {2r} {k}) ^ k $

Hadamard Lora: Activation

$ Delta = odot_ {i = 1} ^ {i = k} ( tanh (u_iv_i ^ t)) $

$ Delta = odot_ {i = 1} ^ {i = k} ( sigma (u_iv_i ^ t)) $

Dylora:

Mettez à jour au hasard une série de rangs

Mettre à jour une partie des calques

Référence:

 @online { kexuefm-9590 ,
    title = {梯度视角下的LoRA：简介、分析、猜测及推广} ,
    author = {苏剑林} ,
    year = { 2023 } ,
    month = { Apr } ,
    url = { url{https://spaces.ac.cn/archives/9590} } ,
}

 @misc { hyeonwoo2023fedpara ,
      title = { FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning } , 
      author = { Nam Hyeon-Woo and Moon Ye-Bin and Tae-Hyun Oh } ,
      year = { 2023 } ,
      eprint = { 2108.06098 } ,
      archivePrefix = { arXiv } ,
      primaryClass = { cs.LG }
}