The goal of this project is to create quality ML Hokkien Translations.
This project contains tools to help translate and evaluate English, Hokkien (POJ script), Hokkien (Tai-Lo script), and Hokkien (漢字 script).
This project focuses on Text-to-Text translations.
(Hokkien is also known as Minnan, Taiwanese, Hoklo, Southern Min, and iso 639-3: NAN.)
Demo
Try out the Online Demo of the latest Hokkien Translation Model
Updates
2023-11-07
Added Models, Translations, and Evaluations of Hokkien (Latin script) -> Hokkien (漢字 script)*
Hokkien (Latin Script) = A mix of Manual and Automated Translations/Transliterations. Automated ones are a mix of Southern + Northern Hokkien Dialects, and also a mix of Tai-Lo and POJ scripts.
Results: Fine Tuned GPT3.5 achieved 30% BLEU (5x more than GPT4-Zero-Shot which got 6%).
Outcomes: This model would be useful for processing Hokkien Wikipedia, as it is the largest source of easily accessible Hokkien Texts.
2023-10-31
Data Management; Moved basic MoeDict transformations to dbt Pipeline. Appended MoeDict data for Mart_sample usecases as well.
2023-10-26
Added Translations and Evaluations of: GPT-3.5 Fine Tuned on 12,000 Examples (Almost all of MoeDict samples), for Mandarin -> Hokkien (漢字 script).
Result: BLEU score of 21
Conclusions:
A Finetuned GPT3.5 Model definitely performs better than a GPT4 Zero-Shot Model when there is 1000+ sentence-pairs.
A Finetuned GPT3.5 Model with ~10,000 sentence pairs performs ~ ↑ 55% better than GPT4 Zero-Shot, and ~ ↑ 282% better than GPT3.5 Zero-Shot.
2023-10-24
Added MoeDict Dataset. It along with an "English" column (translated from Mandarin via GPT4).
Calculated BLEU Scores with new data.
️ Discovered previous BLEU Score calculations were off. Update with Corrected BLEU Scores!
(Data Structures: Refactored so they're easier to deal with.)
Findings:
English -> Hokkien (POJ script) - Initial naieve models have very low BLEU scores (1%)
Mandarin -> Hokkien (漢字 script) - Has a much higher BLEU (7% to 17%). This is about half of what one would expect of a passable BLEU score (30%).
GPT-3.5 Zero Shot: BLEU 7%
GPT-3.5 Fine Tuned on 100 Examples: 10%
GPT-4 Zero Shot: BLEU 13%
GPT-3.5 Fine Tuned on 1,000 Examples: 16%
(Yup, a Fine-Tuned GPT3.5 Model surpases GPT-4 Zero Shot)
Hypotheses:
For ZH->NAN(TC): given the change in Magnitude with Finetuning (0->100->1,000 Examples = 7%->10%->16% BLEU), it is foreseeable that if most of the MOEDICT dataset is used (~13,201 sentence pairs), then there's a chance the BLEU score could reach a passable level (30%).
2023-10-19
Management: Continuing to replace more data models, with dbt models.
2023-10-12
Management: Formatted the downstream 'ML_TESTSET_EVALUATIONS_AVERAGE' table as a dbt Model, as part of the pipeline.
2023-10-11
Management: Reformatted the data as SQLITE3, and initialized a DBT Project from it.
2023-10-10
Reference Texts
Gathered some reference text from Wikipedia (GFDL license) and Omniglot (Non-Commercial license)
Cleaned up reference texts
Generated some reference English translations from Minnan Wikipedia (POJ). Generated by taking the "median text" from GPT4 translations. This is not necessarily accurate, but serves as a basis.
Candidate Texts
Generated some EN→NAN translations (via GPT4 and GPT3.5)
Evaluations
Generated several evaluations based on BLEU
Conclusions and Next Steps
Results: The BLEU scores for these evaluations are quite bad, with only unigram scores showing any non-zero results. Things to try to improve this:
A more lenient POJ Tokenizer that tokenizes by Syllable rather than Word. This is because word-separation isn't always consistent.
A more lenient POJ Tokenizer that ignores diacritics. This is because current POJ sources can be inconsistent.
Using Hanzi as a base script before any POJ conversions, for early translation models.
Using Mandarin Chinese as an intermediary.
Consider the use of Tâi-lô (as a Hanzi→Tâi-lô converter currently exists, but not a Hanzi→POJ one). And how Tâi-lô effects some of the source data.
Refer to romanized words, like "Hanzi", as "Hàn-jī / Hàn-lī" in any LLM prompts. Using Hokkien scripts may slightly bias the LLM towards more accurate Hokkien vocabulary, grammar, and script writing.
Pipeline: These were all generated in spreadsheets. In future, they should be better automated as part of a data pipeline.
