FunpySpiderSearchEngine

Word2vec personalized search implementation + Scrapy2.3.0 (crawl data) + ElasticSearch7.9.1 (storing data and providing external Restful API) + Django3.1.1 search

This warehouse is the ElasticSearch code for crawler data storage . To realize the entire search, it is necessary to combine the Django website project https://github.com/mtianyan/mtianyanSearch

Available features:

1. Zhihu Answer Question Crawler Save ElasticSearch
2. Full text search (need to be used together with the website), the search terms are highlighted and marked with red
3. The real-time three-station crawled number displayed by Redis, Top-5 popular searches
4. word2vec changes ElasticSearch(function_score, script_score) score. For example, if you searched Apple in history, it will make Apple and Jobs ranked high in keyword scoring calculated by Word2vec.

For the entire process of word2vec model training, please check the use of README word2vec in Word2VecModel, which affects the scoring of ElasticSearch. Please check the relevant code in mtianyanSearch.

Core scoring code:

 "source": "double final_score=_score;int count=0;int total = params.title_keyword.size();while(count < total) { String upper_score_title = params.title_keyword[count]; if(doc['title_keyword'].value.contains(upper_score_title)){final_score = final_score+_score;}count++;}return final_score;"

The score is doubled for each title.

Project demonstration diagram:

How to get started?

1. Install ElasticSearch 7.9.1, (optional configuration of ElasticSearch-head)
2. Configuring the ElasticSearch-analysis-ik plugin
3. Install Redis

This machine runs

 git clone https://github.com/mtianyan/FunpySpiderSearchEngine
# 修改config_template中配置信息后重命名为config.py
# 执行 sites/zhihu/es_zhihu.py

cd FunpySpiderSearchEngine
pip install -r requirements.txt
scrapy crawl zhihu

Docker runs

 docker network create search-spider
git clone https://github.com/mtianyan/mtianyanSearch.git
cd mtianyanSearch
docker-compose up -d
git clone https://github.com/mtianyan/FunpySpiderSearchEngine
cd FunpySpiderSearchEngine
docker-compose up -d

Visit 127.0.0.1:8080

sponsor

If my project code is helpful to you, please give me a bun of spicy strips!