bert_seq2seq

bert_seq2seq

A lightweight small frame, welcome to star if you like~ Thank you. If you encounter any problems, you can also ask an issue and you will be sure to reply.

Currently, a version of distributed training is refactored. You can directly train multi-GPUs without additional commands or additional code ! Welcome to https://github.com/920232796/bert_seq2seq_DDP for more information

Welcome to join the communication group~ You can ask questions, give suggestions, and communicate with each other QQ group: 975907202

This framework can currently do various NLP tasks, and the supported models are:

1. bert
2. roberta
3. roberta-large
4. gpt2
5. t5
6. Huawei nezha model
7. bart-Chinese

Supported tasks are:

1. seq2seq For example, writing poems, couplets, automatic titles, automatic abstracts, etc.
2. cls_classifier classifies by extracting the cls vector at the beginning of the sentence, such as sentiment analysis, text classification, semantic matching, etc.
3. sequence_labeling Sequence labeling tasks, such as naming entity recognition, part-of-speech annotation, Chinese word participle, etc.
4. sequence_labeling_crf Add CRF Loss' sequence labeling task to better effect.
5. relation_extract relation extraction, such as triple extraction task. (The example of Mr. Su Jianlin is not exactly the same.)
6. The simbert SimBert model generates similar sentences and makes similarity judgments for similar sentences.
7. multi_label_cls Multi-label classification.

Loading different models is achieved by setting the "model_name" parameter, and different tasks are achieved by setting the "model_class" parameter. For details, you can see various examples in examples.

Summary of the download address of pre-trained model:

1. The roberta model, model and dictionary files need to be downloaded here at https://drive.google.com/file/d/1iNeYFhCBJWeUsIlnW_2K6SMwXkM4gLb_/view. For details, you can refer to this github repository~ https://github.com/ymcui/Chinese-BERT-wwm, and the roberta-large model can also be downloaded in it.
2. bert model (large is not currently supported), download bert Chinese pre-training weight "bert-base-chinese": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-pytorch_model.bin", download bert Chinese dictionary "bert-base-chinese": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt".
3. nezha model, dictionary weight position (currently only supports base): nezha-base model download: link: https://pan.baidu.com/s/1Z0SJbISsKzAgs0lT9hFyZQ Extraction code: 4awe
4. gpt2 model, you can view the gpt_test file in the test file for text rewriting test. The download address of gpt2 Chinese general model and dictionary: https://pan.baidu.com/s/1vTYc8fJUmlQrre5p0JRelw Password: f5un
5. The gpt2 English model is specifically used to refer to the pre-trained model https://huggingface.co/pranavpsv/gpt2-genre-story-generator. For the specific training code, you can see gpt2_english_story_train.py in the example
6. It supports the t5 model, both in English and Chinese, and directly uses the transformers package to load. For details, you can see the relevant examples in the examples folder. Pre-training parameters download: https://github.com/renmada/t5-pegasus-pytorch
7. The SimBert model supports the generation of similar sentences, and the pre-trained model can be used with bert, roberta, and nezha.
8. Bart Chinese model download address: https://huggingface.co/fnlp/bart-base-chinese

Some of the codes are referenced https://github.com/huggingface/transformers/ and https://github.com/bojone/bert4keras Thanks very much! ! !

Screenshots of effects of some small examples

gpt2 generation

enter:

Good weather today

Output:

, I went there to watch a movie with my babies, it was really good! The environment is nothing to say. The movie is very exquisite and the sound effects are very good. I don’t know if this store is still open. I hope you can check it out frequently if you have time.

Writing a poem

bert+crf ner

enter: Output:

News Summary Text Classification (14 categories)

Output:

Medical ner

enter:

Drug interactions may occur if used concurrently with other drugs. Please consult your physician or pharmacist for details. Take it with boiling water, 14 grams at a time, 3 times a day. Nourish blood, regulate menstruation, and relieve pain. It is used for low menstrual flow, wrong menstrual abdominal pain, Jianmin Group Yekai Thai Medicine (Suizhou) Co., Ltd. 1. Avoid eating raw and cold foods. 2. People with other diseases should take it under the guidance of a doctor. 3. If you have normal menstruation, you should go to the hospital for treatment after a sudden decrease in menstruation or have a wrong menstruation period. 4. For the treatment of dysmenorrhea, it is advisable to take the medicine 3 to 5 days before menstruation and take it for one week. If there are fertility requirements, you should take it under the guidance of a doctor. 5. Those who do not relieve dysmenorrhea after taking the medicine or who have severe dysmenorrhea should go to the hospital for diagnosis and treatment. 6. If the symptoms do not relieve after taking the medicine for 2 weeks, you should go to the hospital for treatment. 7. It is prohibited for those who are allergic to this product, and it is used with caution if you are allergic to the body. 8. It is prohibited to use when the characteristics of this product change. 9. Please place this product in a place that children cannot reach. 10. If you are using other drugs, please consult your doctor or pharmacist before using this product. This product is an over-the-counter drug for gynecological irregular menstruation. Nourish blood, regulate menstruation, and relieve pain. It is used for abdominal pain during menstruation. Nourish blood, regulate menstruation, and relieve pain. It is used for low menstrual flow, wrong menstrual abdominal pain, 14g*5 bags of over-the-counter drugs (Class B), National Medical Insurance Directory (Class B) forbidden for pregnant women. Diabetics are not allowed to take it.

Output:

couplet

Semantic Matching

Participle

Install

1. Install this framework pip install bert-seq2seq
2. Install pytorch
3. Install tqdm can be used to display progress bar pip install tqdm
4. Prepare your own data, just modify the read_data function in the example code, construct the input and output, and then start training.
5. Go to the example folder and run the corresponding *_train.py file. For different tasks, run different train.py files. You need to modify the structure of the input and output data and then conduct training. For details, you can see the various examples in the examples ~

Some function explanations

def load_bert(word2ix, model_name="roberta", model_class="seq2seq")

Loading the bert model, the model_name parameter specifies which bert to use, currently supports bert, roberta, and nezha; model_class specifies which task to use bert to do, seq2seq represents the generation task, cls represents the text classification task...

model.load_pretrain_params(pretrain_model_path)

Loading bert model parameters, note that it is only loading the encoder parameters, that is, the parameters of the pre-trained model downloaded from the Internet; for example, the seq2seq model includes the parameters of the bert model + the full connection layer, and this function only loads the first part of the parameters.

def model.load_all_params(recent_model_path)

Load all model parameters. After you have trained for some time and saved the model, you can load the last model training results through this function, continue training or test.

If you want to read various articles, you can go to my website~ http://www.blog.zhxing.online/#/ Search for poems, couplets, NER or news abstract text classification to find the corresponding article. Thank you for your support.

Update records

2021.11.12: Optimized code, supporting the roberta-large model.

2021.10.12: The decoding method of ner has been optimized. There were bugs in the previous coarse-grained decoding method.

2021.08.18: A large amount of code has been optimized. The framework code currently looks clearer and a large amount of redundant code has been deleted.

2021.08.17: It supports Huawei's nezha model. It is very simple. Just change the model_name parameter. Welcome to test the effect.

2021.08.15: Added an example of word segmentation, and added rematch code to tokenizer.

2021.07.29: Optimizing some code is more concise.

2021.07.20: The SimBert model is reproduced, and the output of similar sentences can be performed, but due to the small amount of data, it remains to be tested.

2021.03.19: Supports model extension, and not only uses the model that comes with the framework, but also can directly load the model on the hugging face for training and prediction.

2021.03.12: Added an example of gpt2 Chinese training, Zhou Gong’s dream interpretation.

2021.03.11: Added gpt2 example, you can continue writing the article.

2021.03.11: A random generation decoding method has been added, and the generation has become more diverse.

2021.03.08: beam search returns n results, taking one randomly as output.

2021.02.25: Added an example of semantic matching.

2021.02.06: The device setting method has been adjusted, and it is now more convenient.

2021.1.27: The code structure of the framework has been adjusted and there are many changes. If there is a bug, please feel free to issue.

2021.1.21: Added a new example, character relationship extraction classification.

2020.12.02: Some code has been adjusted and several test files have been added, which can easily load the trained model and test the corresponding tasks.

2020.11.20: Added an example, triple extraction f1 can currently reach 0.7. Added test code for news summary text classification.

2020.11.04: An example of running bet-crf to do ordinary ner tasks, the effect is good.

2020.10.24: A large amount of code has been adjusted and an automatic summary example of the THUCNews dataset has been added. Now, the training should have been very effective. In the past, the pre-training parameters may not be loaded, and the effect may be very poor.

2020.10.23: Some code structure has been adjusted, some variables in each example have been written as global variables, and the beam-search code has been changed, which has been more streamlined. However, the rhymes in writing poems are not supported for the time being. Make up for it later.

2020.09.29: A new training example of Tianchi Medical Ner competition (Medical Ner_train.py) is added. For details, please see the competition interface: https://tianchi.aliyun.com/competition/entrance/531824/information

2020.08.16: Added an example of joint training of poetry couplets (poetry couplets_train.py), you can write poetry and lyrics and couplets at the same time; in addition, a new test code for poetry is added, and you can test it after the model is trained.

2020.08.08: There is a lot of content in this update, 1. Added an example of automatic summary (auto_title.py) 2. Added a code for a simplified vocabulary list. The original 3W words were reduced to more than 1W (because some words will never appear) 3. Modified some beam-search code, and the effect was better. 4. The fine-grained ner cannot be used for the time being, and the data is a bit problematic, so put it in the test folder temporarily. If you find the right data, you can use it 5. Add a new test folder. The trained model can be tested in it to see the effect.

2020.06.22: Supplemented an article by Conditional Layer Norm. Some of the code was explained. http://www.blog.zhxing.online/#/readBlog?blogId=347

2020.06.21: A lot of code has been updated and an example of triple extraction has been reproduced (triple extraction_train.py)~

2020.06.02: I have been busy with graduation recently, and there is still a competition. I won’t update it for the time being, and I will continue to update it in the future.

2020.04.18: After training the bert+crf model, the learning rate of the crf layer does not seem to be high enough and needs to be improved (the crf layer learning rate can now be set separately, generally set to 0.01).

2020.04.13: Added NER task + CRF layer Loss, running through training examples, but the Viterbi algorithm has not been added yet.

2020.04.11: Plan to add a CRF layer to the NER task.

2020.04.07: Added an example of ner.

2020.04.07: Updated pypi and added a model for sequence annotation tasks such as ner.

2020.04.04: The code above pypi has been updated. Currently, the latest version is 0.0.6. Please use the latest version, there will be fewer bugs.

2020.04.04: Fixed some bugs and added examples of news title text classification

2020.04.02: The degree of punishment for repeated words and rhymes in beam-search for writing poems may be better.

2020.04.02: Added Zhou Gong’s dream interpretation task

2020.04.02: Added a task of couplets

2020.04.01: Added a task for writing poetry

2020.04.01: Refactored the code and it took less time to start training a new task.

python setup.py sdist twin upload dist/bert_seq2seq-2.3.5.tar.gz