kcws

Quote

BiLSTM+CRF reference paper for this project model: http://www.aclweb.org/anthology/N16-1030, IDCNN+CRF reference paper: https://arxiv.org/abs/1702.02098

Build

1. Install the bazel code construction tool and install tensorflow (currently, this project requires tf 1.0.0alpha version or above)

2. Switch to the code directory of this project and run ./configure

3. Compile backend services

   bazel build //kcws/cc:seg_backend_api

train

1. Follow the official account of Kaizi Jizhong to reply to kcws to get the corpus download address:

   

2. Unzip the corpus into a directory

3. Switch to the code directory and run:

python kcws/train/process_anno_file.py <corpus directory> pre_chars_for_w2v.txt

bazel build third_party/word2vec:word2vec

Get the preliminary word list first

./bazel-bin/third_party/word2vec/word2vec -train pre_chars_for_w2v.txt -save-vocab pre_vocab.txt -min-count 3

Process low-frequency words python kcws/train/replace_unk.py pre_vocab.txt pre_chars_for_w2v.txt chars_for_w2v.txt

Training word2vec

./bazel-bin/third_party/word2vec/word2vec -train chars_for_w2v.txt -output vec.txt -size 50 -sample 1e-4 -negative 5 -hs 1 -binary 0 -iter 5

Building training corpus tools

bazel build kcws/train:generate_training

Generate materials

./bazel-bin/kcws/train/generate_training vec.txt <corpus directory> all.txt

Get the train.txt , test.txt file

python kcws/train/filter_sentence.py all.txt

4. Install tensorflow, switch to the kcws code directory, and run:

python kcws/train/train_cws.py --word2vec_path vec.txt --train_data_path <absolute path to train.txt> --test_data_path test.txt --max_sentence_len 80 --learning_rate 0.001 (The default IDCNN model is used, you can set the parameter "--use_idcnn False" to switch the BiLSTM model)

5. Generate vocab

bazel build kcws/cc:dump_vocab

./bazel-bin/kcws/cc/dump_vocab vec.txt kcws/models/basic_vocab.txt

6. Export the trained model

python tools/freeze_graph.py --input_graph logs/graph.pbtxt --input_checkpoint logs/model.ckpt --output_node_names "transitions,Reshape_7" --output_graph kcws/models/seg_model.pbtxt

7. Download part-of-speech annotation model (temporary solution, subsequent documents provide part-of-speech annotation model training, export, etc.)

   Download pos_model.pbtxt from https://pan.baidu.com/s/1bYmABk to the kcws/models/ directory

8. Run the web service

./bazel-bin/kcws/cc/seg_backend_api --model_path=kcws/models/seg_model.pbtxt(absolute path to seg_model.pbtxt>) --vocab_path=kcws/models/basic_vocab.txt --max_sentence_len=80

Training instructions for part-of-speech annotation:

https://github.com/koth/kcws/blob/master/pos_train.md

Custom Dictionary

Currently, the custom dictionary is supported in the decoding stage. For specific usage methods, please refer to the kcws/cc/test_seg.cc dictionary is in text format, and each line format is as follows:

<Custom entry>t<weight>

for example:

Blue Skinny Mushroom 4

The weight is a positive integer, generally more than 4, the larger the more important

demo

http://45.32.100.248:9090/

Appendix: Use the same model to identify the company name to train:

http://45.32.100.248:18080