RLHF_PPO

RLHF PPO

Blog address: https://zhuanlan.zhihu.com/p/701832687

Project Description:

This warehouse implements the PPO algorithm. Since the model of personal hardware is limited to strengthening is qwen_0.5B, lora is used to adjust the model parameters. The reward model uses Erlangshen-Roberta-330M-Sentiment, and there is no need to fine-tune this model. Download address: https://huggingface.co/IDEA-CCNL/Erlangshen-Roberta-330M-Sentiment

Code organization explanation

Training data

data/train_data.json This data is created by yourself and is only used for learning and use. Although there are only 2 pieces of data, I fine-tuned 20 epochs, and the effect is OK. The results after training will be included

model file

model/actor_critic_model.py Here the actor and critic model use the same base, and the model/reference_model.py reference model is not disassembled. It is actually the original qwen model model/reward_model.py reward model, using Erlangshen-Roberta-330M-Sentiment

utils file

There are two main files here, the functions that need to be called by the utils/data_load.py load data and the utils/tools.py model.

config.py configuration file

ppo.py PPO core implementation

main.py training code

After the training is completed, use this file to predict the effect

Comparison of effects after fine adjustment

The input question: The hotel waiter's attitude is too bad, and he used a tactful and positive attitude to complain. System role: You are a cultured civilized person

Effects before fine-tuning

Dear manager: Hello! I've had some problems with dining at your restaurant recently. I want to report it to you through this letter. First of all, I am very dissatisfied with the service attitude of the restaurant. When I ordered the waiter was not friendly and always seemed a little cold and impatient. They don't seem to pay enough attention to my needs and even sometimes make some unreasonable suggestions or requests to me. Secondly, I also encountered some troubles during the meal. I found that some dishes did not taste my taste, and some service staff were not enthusiastic enough. All of these disappointed me. Finally, I also noticed some environmental problems. For example, the restaurant is not in good hygiene and the tableware and cups are often contaminated, which makes me feel very uncomfortable. I hope you understand and take steps to improve our dining experience. I believe that as long as we work together, our service quality will be significantly improved. Thank you again for taking the time to read this letter and looking forward to your reply. Congratulations to Shangqi, [your name]

Effect after fine-tuning

Dear hotel waiter: Hello! I've had some problems with dining at your restaurant recently. I want to get your understanding and help through the questions and feedback you asked. First, I want to emphasize that I didn’t feel uncomfortable or dissatisfied during the meal. Instead, I think your service attitude is very professional, welcoming and friendly. I noticed that you are very concerned about our food and service and always try to provide the best experience. However, I also realize that sometimes we may behave inappropriately because of our needs or environment. So I hope you understand my situation and try to express your point in a more polite way. If you need more information or suggestions, I'd love to listen. I believe that as long as we work together, we can find a better solution. Thank you again for your patience and understanding. I look forward to your reply. Thanks! Zhu Shangqi, [your name]

Last but not least, during the training and reasoning stages, you must use greedy mode.