Unduh parsner - Unduh Kode Sumber parsner

parsner

Kode Sumber AI

1.0.0

Unduh

Parsner?

Perkenalan

Repo ini berisi semua model pretrained yang ada yang disesuaikan untuk tugas pengenalan entitas yang disebutkan (NER). Model -model ini dilatih pada dataset NER campuran yang dikumpulkan dari Arman, PEYMA, dan Wikiann yang mencakup sepuluh jenis entitas:

Tanggal (DAT)
Acara (Hawa)
Fasilitas (FAC)
Lokasi (LOC)
Uang (Sen)
Organisasi (org)
Persen (PCT)
Orang (per)
Produk (Pro)
Waktu (Tim)

Informasi dataset

	Catatan	B-dat	B-EVE	B-fac	BLOK	B-mon	B-org	B-PCT	B-per	B-Pro	B-Tim	I-dat	I-eve	I-fac	I-loc	I-mon	I-org	I-PCT	SAYA PER	I-Pro	I-Tim
Kereta	29133	1423	1487	1400	13919	417	15926	355	12347	1855	150	1947	5018	2421	4118	1059	19579	573	7699	1914	332
Sah	5142	267	253	250	2362	100	2651	64	2173	317	19	373	799	387	717	270	3260	101	1382	303	35
Tes	6049	407	256	248	2886	98	3216	94	2646	318	43	568	888	408	858	263	3967	141	1707	296	78

Unduh Anda dapat mengunduh dataset dari sini

Evaluasi

Tabel berikut merangkum skor yang diperoleh oleh model pretrained secara keseluruhan dan per setiap kelas.

Model	ketepatan	presisi	mengingat	f1
Bert	0.995086	0.953454	0.961113	0.957268
Roberta	0.994849	0.949816	0.960235	0.954997
Distilbert	0.994534	0.946326	0.95504	0.950663
Albert	0.993405	0.938907	0.943966	0.941429

Bert

	nomor	presisi	mengingat	f1
Dat	407	0.860636	0.864865	0.862745
MALAM	256	0.969582	0.996094	0.982659
Fac	248	0.976190	0.991935	0.984000
Loc	2884	0.970232	0.971914	0.971072
Senin	98	0.905263	0.877551	0.891192
Org	3216	0.939125	0.954602	0.946800
Pct	94	1.000000	0.968085	0.983784
PER	2645	0.965244	0.965974	0.965608
PRO	318	0.981481	1.000000	0.990654
Tim	43	0.692308	0.837209	0.757895

Roberta

	nomor	presisi	mengingat	f1
Dat	407	0.844869	0.869779	0.857143
MALAM	256	0.948148	1.000000	0.973384
Fac	248	0.957529	1.000000	0.978304
Loc	2884	0.965422	0.968100	0.966759
Senin	98	0.937500	0.918367	0.927835
Org	3216	0.943662	0.958333	0.950941
Pct	94	1.000000	0.968085	0.983784
PER	2646	0.957030	0.959562	0.958294
PRO	318	0.963636	1.000000	0.981481
Tim	43	0.739130	0.790698	0.764045

Distilbert

	nomor	presisi	mengingat	f1
Dat	407	0.812048	0.828010	0.819951
MALAM	256	0.955056	0.996094	0.975143
Fac	248	0.972549	1.000000	0.986083
Loc	2884	0.968403	0.967060	0.967731
Senin	98	0.925532	0.887755	0.906250
Org	3216	0.932095	0.951803	0.941846
Pct	94	0.936842	0.946809	0.941799
PER	2645	0.959818	0.957278	0.958546
PRO	318	0.963526	0.996855	0.979907
Tim	43	0.760870	0.813953	0.786517

Albert

	nomor	presisi	mengingat	f1
Dat	407	0.820639	0.820639	0.820639
MALAM	256	0.936803	0.984375	0.960000
Fac	248	0.925373	1.000000	0.961240
Loc	2884	0.960818	0.960818	0.960818
Senin	98	0.913978	0.867347	0.890052
Org	3216	0.920892	0.937500	0.929122
Pct	94	0.946809	0.946809	0.946809
PER	2644	0.960000	0.944024	0.951945
PRO	318	0.942943	0.987421	0.964670
Tim	43	0.780488	0.744186	0.761905

Cara menggunakan

Anda menggunakan model ini dengan pipa Transformers untuk NER.

Memasang persyaratan

pip install sentencepiece
pip install transformers

Cara memprediksi menggunakan pipa

 from transformers import AutoTokenizer
from transformers import AutoModelForTokenClassification  # for pytorch
from transformers import TFAutoModelForTokenClassification  # for tensorflow
from transformers import pipeline

# model_name_or_path = "HooshvareLab/bert-fa-zwnj-base-ner"  # Roberta
# model_name_or_path = "HooshvareLab/roberta-fa-zwnj-base-ner"  # Roberta
model_name_or_path = "HooshvareLab/distilbert-fa-zwnj-base-ner"  # Distilbert
# model_name_or_path = "HooshvareLab/albert-fa-zwnj-base-v2-ner"  # Albert

tokenizer = AutoTokenizer . from_pretrained ( model_name_or_path )

model = AutoModelForTokenClassification . from_pretrained ( model_name_or_path )  # Pytorch
# model = TFAutoModelForTokenClassification.from_pretrained(model_name_or_path)  # Tensorflow

nlp = pipeline ( "ner" , model = model , tokenizer = tokenizer )
example = "در سال ۲۰۱۳ درگذشت و آندرتیکر و کین برای او مراسم یادبود گرفتند."

ner_results = nlp ( example )
print ( ner_results )

Model

Memeluk hub model wajah

Bert
Roberta
Distilbert
Albert

Pelatihan

Semua model dilatih pada GPU NVIDIA P100 tunggal dengan parameter berikut.

Argumen

 " task_name " : " ner "
" model_name_or_path " : model_name_or_path
" train_file " : " /content/ner/train.csv "
" validation_file " : " /content/ner/valid.csv "
" test_file " : " /content/ner/test.csv "
" output_dir " : output_dir
" cache_dir " : " /content/cache "
" per_device_train_batch_size " : 16
" per_device_eval_batch_size " : 16
" use_fast_tokenizer " : True
" num_train_epochs " : 5.0
" do_train " : True
" do_eval " : True
" do_predict " : True
" learning_rate " : 2e-5
" evaluation_strategy " : " steps "
" logging_steps " : 1000
" save_steps " : 1000
" save_total_limit " : 2
" overwrite_output_dir " : True
" fp16 " : True
" preprocessing_num_workers " : 4

Mengutip

Harap kutip repositori ini dalam publikasi sebagai berikut:

 @misc{ParsNER,
  author = {Hooshvare Team},
  title = {Pre-Trained NER models for Persian},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {url{https://github.com/hooshvare/parsner}},
}