Unduh haloop - Unduh Kode Sumber haloop

haloop

Kode Sumber AI

Training Transformers

Unduh

Haloop

Haloop adalah alat agen bicara. Haloop menyediakan:

Program hai untuk menginisialisasi model;
Program hac untuk Pelatihan Model Akustik;
har untuk pelatihan dan evaluasi model bahasa RNN;
hal untuk pelatihan model perhatian kausal;
hat untuk pengujian agen;
hap untuk mencetak probabilitas log kalimat di bawah model bahasa GPT;
haw untuk membandingkan label dalam dataset menggunakan tingkat kesalahan kata;
hax untuk menghitung korelasi antar dataset;

Paket dapat diinstal dari PYPI:

 pip install haloop

Model pretrained

hat dapat digunakan dengan model GPT-2 Ukraina dari makalah kami GPT-2 metadata pretraining menuju instruksi finetuning untuk Ukraina.

Anda harus menginstal dan mengunduh:

 pip install bitsandbytes sentencepiece

wget https://a.wilab.org.ua/gpt/wiki.model  # sentencepiece tokenizer
wget https://a.wilab.org.ua/gpt/ckpt10m.pt  # model checkpoint for GPT-2 Large

Sekarang, kick off the repl:

 hat --spm wiki.model ckpt10m.pt

Skor daftar kalimat dengan menghitung probabilitas log di bawah model bahasa. Pertama, file input akan diurutkan berdasarkan jumlah token untuk meningkatkan pemanfaatan GPU:

 cat ubertext.wikipedia.filter_rus_gcld+short.text_only.txt | spm_encode --model wiki.model | awk -v OFS="t" '{ print length, $0 }' | sort -r -n -s | cut -f2-  | spm_decode --model wiki.model > wikipedia.toksorted.txt
cat wikipedia.toksorted.txt | hap --compile --spm wiki.model ckpt10m.pt | pv -l > wikipedia.toksorted.scores.txt

Mengutip

Tolong kutip:

 @inproceedings{kyrylov-chaplynskyi-2023-gpt,
    title = "{GPT}-2 Metadata Pretraining Towards Instruction Finetuning for {U}krainian",
    author = "Kyrylov, Volodymyr  and
      Chaplynskyi, Dmytro",
    booktitle = "Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.unlp-1.4",
    pages = "32--39",
    abstract = "We explore pretraining unidirectional language models on 4B tokens from the largest curated corpus of Ukrainian, UberText 2.0. We enrich document text by surrounding it with weakly structured metadata, such as title, tags, and publication year, enabling metadata-conditioned text generation and text-conditioned metadata prediction at the same time. We pretrain GPT-2 Small, Medium and Large models each on single GPU, reporting training times, BPC on BrUK and BERTScore on titles for 1000 News from the Future. Next, we venture to formatting POS and NER datasets as instructions, and train low-rank attention adapters, performing these tasks as constrained text generation. We release our models for the community at https://github.com/proger/uk4b.",
}