llama2 lora fine tuning

Fine-tune LLaMA2-Chat with Lora and deepspeed

Fine-tune the Llama-2-7b-chat model on two P100s (16G).

The data source adopts the alpaca format and consists of two data sources: train and validation.

1. Graphics card requirements

16G video memory and above (P100 or T4 and above), one or more blocks.

2. Clone source code

git clone https://github.com/git-cloner/llama2-lora-fine-tuning
cd llama2-lora-fine-tuning

3. Installation dependent environment

 # 创建虚拟环境
conda create -n llama2 python=3.9 -y
conda activate llama2
# 下载github.com上的依赖资源（需要反复试才能成功，所以单独安装）
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1
pip install git+https://github.com/PanQiWei/AutoGPTQ.git -i https://pypi.mirrors.ustc.edu.cn/simple --trusted-host=pypi.mirrors.ustc.edu.cn
pip install git+https://github.com/huggingface/peft -i https://pypi.mirrors.ustc.edu.cn/simple
pip install git+https://github.com/huggingface/transformers -i https://pypi.mirrors.ustc.edu.cn/simple
# 安装其他依赖包
pip install -r requirements.txt -i https://pypi.mirrors.ustc.edu.cn/simple
# 验证bitsandbytes
python -m bitsandbytes

4. Download the original model

python model_download.py --repo_id daryl149/llama-2-7b-chat-hf

5. Expand the Chinese word list

 # 使用了https://github.com/ymcui/Chinese-LLaMA-Alpaca.git的方法扩充中文词表
# 扩充完的词表在merged_tokenizes_sp（全精度）和merged_tokenizer_hf（半精度）
# 在微调时，将使用--tokenizer_name ./merged_tokenizer_hf参数
python merge_tokenizers.py 
  --llama_tokenizer_dir ./models/daryl149/llama-2-7b-chat-hf 
  --chinese_sp_model_file ./chinese_sp.model

6. Fine-tuning parameter description

There are several parameters that can be adjusted:

parameter	illustrate	Get the value
load_in_bits	Model accuracy	4 and 8. If the video memory does not overflow, try to choose high-precision 8.
block_size	Maximum length of token	First choice 2048, memory overflow, 1024, 512, etc.
per_device_train_batch_size	Number of batches per card loaded at each time during training	As long as the memory does not overflow, try to go to the general election
per_device_eval_batch_size	Number of batches per card loaded at each time during evaluation	As long as the memory does not overflow, try to go to the general election
include	Graphics Card Sequences Used	For example, two pieces: localhost:1,2 (note that the sequence is not necessarily the same as what nvidia-smi sees)
num_train_epochs	Number of training rounds	At least 3 rounds

7. Fine adjustment

chmod +x finetune-lora.sh
# 微调
./finetune-lora.sh
# 微调（后台运行）
pkill -9 -f finetune-lora
nohup ./finetune-lora.sh > train.log  2>&1 &
tail -f train.log

8. Test

CUDA_VISIBLE_DEVICES=0 python generate.py 
    --base_model ' ./models/daryl149/llama-2-7b-chat-hf ' 
    --lora_weights ' output/checkpoint-2000 ' 
    --load_8bit #不加这个参数是用的4bit