llama-lora-chinese-couplet
A simple llama-lora end to end example to show its potential to train a Chinese Couplet AI
- int8 LLaMa + LoRA = Finetune in a consumer GPU <= 10G vram!
Slides, last update 04/29/2023
- LLM, Pretrain, Finetune
- LoRA: A popular Parameter Efficient Finetune approach by Microsoft
- LLaMA: A gift from Facebook to the research community
- LLaMA-2: A newer and better gift from Facebook
- Alpaca: A $600 approach to “distill 80%” of ChatGPT by Stanford
- Alpaca-LoRA: An even cheaper approach from a Stanford student？
- Omitted due to time
- Quantization intro, LLM.int8
Zero-shot Examples
- after 3 epochs of 5k pairs, cap max tokens, greedy
- post-processing to match # of chinese chars (so yes, I cheated ^_^)
- ideally a well trained model will know end of sentence (eos) itself
- prompt:
对联：{上联}n下联：
| 上聯 | Base LLaMA | LLaMa_LoRA_A100_9mins | LLaMa_LoRA_Tesla_T4_35mins |
|---|
| 春風得意花鋪路 | 沉浸落泥n上聯 | 月光聽聲風吹夢 | 風雨吹夢浮浮� |
| 美麗中國魅力北京 | 美麗中國魅力北京n上聯： | 歷史濃淺中華夢境 | 夢幻中國夢想宏碁 |
| 魚書千里夢 | 魚肉燒肉n | 鳥聲萬里聲 | 鳥聲萬里聲 |
| 日落晚霞臨古寺 | 晚霞臨古寺n上 | 月映晨雨滿夢境 | 月映晨霜滿夢境 |
- In case you are into Chinese couplets, I have a better T5 version
Some additional thoughts on high-quality AI
