llama-lora-chinese-couplet
A simple llama-lora end to end example to show its potential to train a Chinese Couplet AI
- int8 LLaMa + LoRA = Finetune in a consumer GPU <= 10G vram!
Slides, last update 04/29/2023
- LLM, Pretrain, Finetune
- LoRA: A popular Parameter Efficient Finetune approach by Microsoft
- LLaMA: A gift from Facebook to the research community
- LLaMA-2: A newer and better gift from Facebook
- Alpaca: A $600 approach to “distill 80%” of ChatGPT by Stanford
- Alpaca-LoRA: An even cheaper approach from a Stanford student？
- Omitted due to time
- Quantization intro, LLM.int8
Zero-shot Examples
- after 3 epochs of 5k pairs, cap max tokens, greedy
- post-processing to match # of chinese chars (so yes, I cheated ^_^)
- ideally a well trained model will know end of sentence (eos) itself
- prompt:
对联：{上联}n下联：
| 上联 |
Base LLaMA |
LLaMa_LoRA_A100_9mins |
LLaMa_LoRA_Tesla_T4_35mins |
| 春风得意花铺路 |
沉浸落泥n上联 |
月光听声风吹梦 |
风雨吹梦浮浮� |
| 美丽中国魅力北京 |
美丽中国魅力北京n上联： |
历史浓浅中华梦境 |
梦幻中国梦想宏碁 |
| 鱼书千里梦 |
鱼肉烧肉n |
鸟声万里声 |
鸟声万里声 |
| 日落晚霞临古寺 |
晚霞临古寺n上 |
月映晨雨满梦境 |
月映晨霜满梦境 |
- In case you are into Chinese couplets, I have a better T5 version
Some additional thoughts on high-quality AI
