MLLM Finetuning Demo

MLLM-Finetuning-Demo

安裝LLaMA-Factory

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[torch,metrics]
cd .. # 回到项目根目录

預訓練

LLaVA中的特徵對齊，凍結language_model和vision_tower ，只微調multi_modal_projector 。

CUDA_VISIBLE_DEVICES=0 llamafactory-cli train config/llava_pt.yaml

微調

CUDA_VISIBLE_DEVICES=0 llamafactory-cli train config/llava_lora_sft.yaml

網頁聊天

CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat 
--model_name_or_path llava-hf/llava-1.5-7b-hf 
--adapter_name_or_path saves/llava1_5-7b/lora/sft 
--template llava

上傳數據集到Huggingface

請在upload_dataset.py中替換您自己的key.

python3 upload_dataset.py

導出和上傳模型到huggingface

請在config/llava_lora_sft_export.yaml中替換您自己的export_hub_model_id和hf_hub_token .

CUDA_VISIBLE_DEVICES=0 llamafactory-cli export config/llava_lora_sft_export.yaml