lightNLP

lightNLP, a very basic natural language processing framework

Introduction

This project is based on Pytorch and torchtext, and aims to provide a basic deep learning framework for natural language processing-related tasks.

For detailed instructions and tutorials, please refer to the project documentation: lightnlp-cookbook

statement

1. In terms of positioning, this project is just a collection and attempt, and its purpose is not used as an enterprise-level and production-level. The target group is mainly computer developers and beginners who are interested in the practice of various tasks of natural language processing, and more importantly, they are self-entertainment .
2. If users or developers who have real scenario needs can refer to Industry Chinese NLP commercial services to seek commercial services, of course, I am willing to provide paid services.
3. For those who have a certain understanding of Pytorch and natural language processing, if you want to pursue rapid development and freely customize nlp applications, you can consider fastNLP open sourced by Fudan University's nlp laboratory, which has rich framework functions and is simple and easy to use.
4. This project does not provide some training data and trained models for each task like some other frameworks so that it can be downloaded and used directly.
5. Many of the models of this project are based on the original implementation on Github and then processed on the basis of the process. Here I would like to express my sincere gratitude to the relevant authors!
6. The various parameters of each task model are not finely tuned, but only to the extent that they can run.
7. This project can only be achieved under the following two development environments. I am not responsible for any problems arising from other environments.
   • Windows 10, Python 3.6, Pytorch 1.3
   • Manjaro, Python 3.7, Pytorch 1.3

Install

pip install lightNLP

It is recommended to use domestic sources to install, such as using the following command:

pip install -i https://pypi.douban.com/simple/ lightNLP

Installation dependencies

Since some libraries such as pytorch and torchtext are not in the pypi source or only have older versions, we need to install some libraries separately.

Install pytorch

Please use the latest version of Pytorch!

For specific installation, please refer to the pytorch official website to select the version that suits you according to the platform, installation method, Python version, and CUDA version.

Install torchtext

Use the following command to install the latest version of torchtext:

pip install https://github.com/pytorch/text/archive/master.zip

Example

Named entity recognition (ner)

1. Training data

BIO

The training data examples are as follows:

清 B_Time
明 I_Time
是 O
人 B_Person
们 I_Person
祭 O
扫 O
先 B_Person
人 I_Person
， O
怀 O
念 O
追 O
思 O
的 O
日 B_Time
子 I_Time
。 O

正 O
如 O
宋 B_Time
代 I_Time
诗 B_Person
人 I_Person

2. Use examples

1. Training

 from lightnlp.sl import NER

# 创建NER对象
ner_model = NER()

train_path = '/home/lightsmile/NLP/corpus/ner/train.sample.txt'
dev_path = '/home/lightsmile/NLP/corpus/ner/test.sample.txt'
vec_path = '/home/lightsmile/NLP/embedding/char/token_vec_300.bin'

# 只需指定训练数据路径和TensorBoard日志文件路径，预训练字向量可选，开发集路径可选，模型保存路径可选（模型保存路径默认为`xx_saves`，其中xx为模型简称，如ner）。
ner_model.train(train_path, vectors_path=vec_path, dev_path=dev_path, save_path='./ner_saves', log_dir='E:/Test/tensorboard/')

2. Test

 # 加载模型，默认当前目录下的`ner_saves`目录
ner_model.load('./ner_saves')
# 对train_path下的测试集进行读取测试
ner_model.test(train_path)

3. Prediction

 from pprint import pprint

pprint(ner_model.predict('另一个很酷的事情是，通过框架我们可以停止并在稍后恢复训练。'))

Prediction results:

 [{'end': 15, 'entity': '我们', 'start': 14, 'type': 'Person'}]

4. Check the training effect

Execute the following command from the command line, where E:TesttensorBoard is modified to be the log storage path during model training, and the port specification is optional:

tensorboard --logdir=E: T est t ensorBoard --port=2019

You can see similar effects:

5. Deploy the service

 ner_model . deploy ( host = "localhost" , port = 2020 , debug = False )

All parameters are optional. host parameter is default to localhost . port port will be automatically applied for an idle port to the system by the program, and debug mode will not be enabled by default.

You can use Postman or write a program to test it, as shown in the figure below:

todo

business

• Add a brief project description
• Provide demo training data for each task

project

• Reconstruct the project structure, merge the same redundant places, and keep the project structure clear
• Added TensorBoard visualization functions, mainly including scalar of loss and score and graph of each model (there are currently some bugs in SummaryWriter's add_graph function in Pytorch, so it cannot be added for the time being.).
• Added simple flask-based model deployment function (currently only used for model training effect testing)
• Now the path and name saved by the model are the same by default and will conflict. Next, each model has its own name .
• Added breakpoint retraining function.
• Add earlyStopping.

Function

• Add CBOW word vector related models and train prediction code
• Add skip_gram related models and training prediction code
• Add sentiment analysis-related models and training prediction codes
• Add text implicit related models and train prediction code
• Add text generation related models and training prediction code
• Add language model-related models and training prediction code
• Add dependency analysis related models and training prediction code
• Add relationship extraction related models and train prediction codes
• Add Chinese word segmentation related models and training prediction codes
• Add part-of-speech annotation related models and training prediction codes
• Add event extraction related models and training prediction code
• Add automatic summary related models and training prediction code
• Add machine translation related models and training prediction code
• Increase sentence similarity correlation model and train prediction code
• Add sequence to sequence-related models and train prediction codes
• Add chatbot-related models and training prediction codes
• Add named entity recognition related models and predictive training code
• Add some transformer implementations for tasks (based on the official implementation of Pytorch)
• Add Elmo related models and training prediction code
• Add GloVe-related models and training prediction code
• Add GPT-related models and training prediction codes
• Add Bert-related models and training prediction codes
• Add attribute extraction related models and train prediction code
• Adding reference digestion related models and training prediction code
• Add word meaning disambiguation related models and train prediction codes
• Add reading comprehension-related models and training prediction codes
• Add keyword extraction related models and training prediction codes
• Add component syntax analysis related models and prediction training codes

Reward

If this project is helpful to you, please give me a reward~