ORPO

ORPO algorithm manually implemented

Blog address: https://zhuanlan.zhihu.com/p/703169002

Project Description:

This warehouse implements the ORPO algorithm. Since the model of personal hardware is qwen_0.5B, lora is used to adjust the model parameters.

Code organization explanation

Training data

data/train_data.json This data is created by yourself and is only used for learning and use. Although there are only 2 pieces of data, I fine-tuned 20 epochs, and the effect is OK. The results after training will be included

model.py The policy model that needs to be optimized here

data_load.py loads data

config.py configuration file

orpo.py orpo core implementation

main.py training code

After the training is completed, use this file to predict the effect

Comparison of effects after fine adjustment

Entered question: Operating Instructions: Design a logo for a website about recycling. Instruction: The following is an explanation of the task. Write a response that properly completes the request.

Effects before fine-tuning

Dear users, thank you for your support! To better meet your needs, we have designed a logo for you. First, we need to determine what the theme of this logo is. If it is an environmentally friendly theme, we can choose colors such as green, blue or white; if it is a recycling theme, we can choose colors such as black, gray or brown, etc. Then we need to consider the size and position of the flag. Generally speaking, the logo should be located in the center of the website so that users can quickly find and use it. Next, we need to consider the design elements of the logo. We can use some simple lines to represent recycling, such as the shape, color, etc. of recycling. At the same time, we can also add some details, such as recycled symbols or recycled patterns, to increase the visual appeal of the logo. Finally, we need to ensure the clarity and legibility of the logo. We should try to avoid using too complex fonts or graphics to avoid confusing users. Hope these suggestions help you design a sign of success. If you have any questions, feel free to ask me.

Effect after fine-tuning

Reply: I'm happy to design a website for you about recycling, but I need more information to complete this project. For example, which color or font do you want to use? Are there any specific ideas or patterns? If you have any questions or need further help, please let me know.

In addition, the target result was unreasonable

Entered question: Operating Instructions: Design a logo for a website about recycling. Instruction: The following is an explanation of the task. Write a response that properly completes the request.

Effect after fine-tuning

I'm sorry, but I can't design a logo for your recycling website. Designing logos requires professional design skills and tools, and I mainly provide text help. It is recommended that you contact a professional graphic designer to complete this task.

Last but not least, during the training and reasoning stages, you must use greedy mode.