DPO

Manual implementation of DPO algorithm

Blog address: https://zhuanlan.zhihu.com/p/702774357

Project Description:

This warehouse implements the DPO algorithm. Since the model of personal hardware is qwen_0.5B, lora is used to adjust the model parameters.

Code organization explanation

Training data

data/train_data.json This data is created by yourself and is only used for learning and use. Although there are only 2 pieces of data, I fine-tuned 3 epochs, and the effect is OK. The results after training will be included

model file

model/model.py The policy model that needs to be optimized here is model/reference_model.py The reference model is actually the original qwen model

utils file

There are mainly two files here, utils/data_load.py loads data

config.py configuration file

dpo.py DPO core implementation

main.py training code

After the training is completed, use this file to predict the effect

Comparison of effects after fine adjustment

Entered question: Operating Instructions: Design a logo for a website about recycling. Instruction: The following is an explanation of the task. Write a response that properly completes the request.

Effects before fine-tuning

Dear users, thank you for your support! To better meet your needs, we have designed a logo for you. First, we need to determine what the theme of this logo is. If it is an environmentally friendly theme, we can choose colors such as green, blue or white; if it is a recycling theme, we can choose colors such as black, gray or brown, etc. Then we need to consider the size and position of the flag. Generally speaking, the logo should be located in the center of the website so that users can quickly find and use it. Next, we need to consider the design elements of the logo. We can use some simple lines to represent recycling, such as the shape, color, etc. of recycling. At the same time, we can also add some details, such as recycled symbols or recycled patterns, to increase the visual appeal of the logo. Finally, we need to ensure the clarity and legibility of the logo. We should try to avoid using too complex fonts or graphics to avoid confusing users. Hope these suggestions help you design a sign of success. If you have any questions, feel free to ask me.

Effect after fine-tuning

Reply: Thank you for your creativity! We are working to create an environmentally friendly and sustainable design that encourages people to take action to protect the environment. We can use green plants, blue or white as the background color and add some recyclable elements to the logo, such as recycling cardboard or bottles.

The more optimized, the worse it is, the training data and resources are limited, for learning reference only

Last but not least, during the training and reasoning stages, you must use greedy mode.