LoRA CLIP

Lora-CLIP：易於洛拉化剪輯的易於包裝器（PYPI）

特徵

1. 跨圖像或文本編碼器或兩個編碼器的動態Lora-Layer插入。

2. 將預訓練的OpenAI夾重負載安全地加入洛拉的模型中。

3. 通過正確凍結positional_embedding ， logit_scale和Project來加載微調現成的模型。

注意：我們目前僅在圖像編碼器的洛拉化學中支持視覺變壓器模型。重新連接模型將很快添加。

安裝

重要的是：在安裝loraclip之前，請確保您有剪輯。這可以通過

pip install regex ftfy
pip install git+https://github.com/openai/CLIP.git

然後，您可以通過命令pip pip install loraclip直接安裝此軟件包。或者，您可以考慮通過

git clone https://github.com/jaisidhsingh/LoRA-CLIP.git
cd LoRA-CLIP
pip install -e .

用法

 import loraclip
import argparse


def test_loraclip ( args ):
	# Easy-to-use with standard CLIP syntax.
	# 1. Dynamic LoRA rank specification
	# 2. Specify which encoder(s) to LoRA-ify
	model , preprocess = loraclip . load ( args . clip_model_name , device = args . device , r = args . lora_rank , lora_mode = args . lora_mode )
	# Utility to preview no. of trainable params along with their % with total params.
	loraclip . print_trainable_parameters ( model )


def setup_args ():
	parser = argparse . ArgumentParser ()
	parser . add_argument ( "--clip-model-name" , type = str , default = "ViT-B/16" )
	parser . add_argument ( "--device" , type = str , default = "cuda" )
	parser . add_argument ( "--lora-rank" , type = int , default = 4 )
	parser . add_argument ( "--lora-mode" , type = str , default = "vision+text" , choices = [ "vision" , "text" , "vision+text" ])

	args = parser . parse_args ()
	return args


if __name__ == "__main__" :
    args = setup_args ()
    test_loraclip ( args )

參考

1. https://github.com/openai/clip

2. https://github.com/sivandoveh/tsvlc