flashtext下載 - flashtext源代碼下載

flashtext

其他源碼

1.0.0

下載

flashtext

該模塊可用於替換句子中的關鍵字或從句子中提取關鍵字。它基於FlashText算法。

安裝

$ pip安裝flashtext

API文檔

可以在flashtext上找到文檔。

用法

提取關鍵字

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> # keyword_processor.add_keyword(<unclean name>, <standardised name>)
>>> keyword_processor.add_keyword( ' Big Apple ' , ' New York ' )
>>> keyword_processor.add_keyword( ' Bay Area ' )
>>> keywords_found = keyword_processor.extract_keywords( ' I love Big Apple and Bay Area. ' )
>>> keywords_found
>>> # ['New York', 'Bay Area']

替換關鍵字

>>> keyword_processor.add_keyword( ' New Delhi ' , ' NCR region ' )
>>> new_sentence = keyword_processor.replace_keywords( ' I love Big Apple and new delhi. ' )
>>> new_sentence
>>> # 'I love New York and NCR region.'

案例敏感的例子

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor( case_sensitive = True )
>>> keyword_processor.add_keyword( ' Big Apple ' , ' New York ' )
>>> keyword_processor.add_keyword( ' Bay Area ' )
>>> keywords_found = keyword_processor.extract_keywords( ' I love big Apple and Bay Area. ' )
>>> keywords_found
>>> # ['Bay Area']

提取的關鍵字跨度

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword( ' Big Apple ' , ' New York ' )
>>> keyword_processor.add_keyword( ' Bay Area ' )
>>> keywords_found = keyword_processor.extract_keywords( ' I love big Apple and Bay Area. ' , span_info = True )
>>> keywords_found
>>> # [('New York', 7, 16), ('Bay Area', 21, 29)]

通過提取的關鍵字獲取額外的信息

>>> from flashtext import KeywordProcessor
>>> kp = KeywordProcessor()
>>> kp.add_keyword( ' Taj Mahal ' , ( ' Monument ' , ' Taj Mahal ' ))
>>> kp.add_keyword( ' Delhi ' , ( ' Location ' , ' Delhi ' ))
>>> kp.extract_keywords( ' Taj Mahal is in Delhi. ' )
>>> # [('Monument', 'Taj Mahal'), ('Location', 'Delhi')]
>>> # NOTE : replace_keywords feature won't work with this.

沒有關鍵字的干淨名稱

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword( ' Big Apple ' )
>>> keyword_processor.add_keyword( ' Bay Area ' )
>>> keywords_found = keyword_processor.extract_keywords( ' I love big Apple and Bay Area. ' )
>>> keywords_found
>>> # ['Big Apple', 'Bay Area']

同時添加多個關鍵字

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_dict = {
>>>     " java " : [ " java_2e " , " java programing " ],
>>>     " product management " : [ " PM " , " product manager " ]
>>> }
>>> # {'clean_name': ['list of unclean names']}
>>> keyword_processor.add_keywords_from_dict(keyword_dict)
>>> # Or add keywords from a list:
>>> keyword_processor.add_keywords_from_list([ " java " , " python " ])
>>> keyword_processor.extract_keywords( ' I am a product manager for a java_2e platform ' )
>>> # output ['product management', 'java']

刪除關鍵字

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_dict = {
>>>     " java " : [ " java_2e " , " java programing " ],
>>>     " product management " : [ " PM " , " product manager " ]
>>> }
>>> keyword_processor.add_keywords_from_dict(keyword_dict)
>>> print (keyword_processor.extract_keywords( ' I am a product manager for a java_2e platform ' ))
>>> # output ['product management', 'java']
>>> keyword_processor.remove_keyword( ' java_2e ' )
>>> # you can also remove keywords from a list/ dictionary
>>> keyword_processor.remove_keywords_from_dict({ " product management " : [ " PM " ]})
>>> keyword_processor.remove_keywords_from_list([ " java programing " ])
>>> keyword_processor.extract_keywords( ' I am a product manager for a java_2e platform ' )
>>> # output ['product management']

在關鍵WordProcessor中檢查術語數量

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_dict = {
>>>     " java " : [ " java_2e " , " java programing " ],
>>>     " product management " : [ " PM " , " product manager " ]
>>> }
>>> keyword_processor.add_keywords_from_dict(keyword_dict)
>>> print ( len (keyword_processor))
>>> # output 4

檢查keywordprocessor中是否存在術語

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword( ' j2ee ' , ' Java ' )
>>> ' j2ee ' in keyword_processor
>>> # output: True
>>> keyword_processor.get_keyword( ' j2ee ' )
>>> # output: Java
>>> keyword_processor[ ' colour ' ] = ' color '
>>> keyword_processor[ ' colour ' ]
>>> # output: color

獲取字典中的所有關鍵字

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword( ' j2ee ' , ' Java ' )
>>> keyword_processor.add_keyword( ' colour ' , ' color ' )
>>> keyword_processor.get_all_keywords()
>>> # output: {'colour': 'color', 'j2ee': 'Java'}

為了檢測單詞邊界，當前除此 w [a-za-z0-9_]以外的任何字符都被視為單詞邊界。

設置或添加字符作為單詞字符的一部分

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword( ' Big Apple ' )
>>> print (keyword_processor.extract_keywords( ' I love Big Apple/Bay Area. ' ))
>>> # ['Big Apple']
>>> keyword_processor.add_non_word_boundary( ' / ' )
>>> print (keyword_processor.extract_keywords( ' I love Big Apple/Bay Area. ' ))
>>> # []

測試

$ git克隆https://github.com/vi3k6i5/flashtext
$ cd flashtext
$ pip安裝pytest
$ python setup.py測試

構建文檔

$ git克隆https://github.com/vi3k6i5/flashtext
$ cd flashtext/docs
$ pip安裝獅身人面像
$製作html
$＃打開_build/html/index.html在瀏覽器中查看本地

為什麼不言論？

這是一種基於Aho-Corasick算法和Trie詞典的自定義算法。

FlashText花費的時間與REGEX相比找到術語。

FlashText與Regex相比，FlashText所花費的時間替換條款。

鏈接到代碼進行基準測試查找功能並替換功能。

該庫的想法來自以下stackoverflow問題。

引用

原始論文發表在FlashText算法上。

 @Article {2017arxiv171100046s，
   作者= {{singh}，V。 }，
    title =“ {更換或檢索文檔中的關鍵字}”，
  日記= {arxiv e-prints}，
ArchivePrefix =“ arxiv”，
   eprint = {1711.00046}，
 primaryclass =“ cs.ds”，
 關鍵字= {計算機科學 - 數據結構和算法}，
     年= 2017年，
    月=十月，
   adsurl = {http://adsabs.harvard.edu/abs/2017arxiv171100046s}，
  adsnote = {由SAO/NASA天體物理數據系統提供
}

這篇文章發表在中等五自由狀運動上。