Verdant_Search

verdant search

Currently, in order to consider the load of the backend, the backend of the search engine has been changed to flask
However, its temporary storage structure CubeQL uses fastapi
Official website: http://115.29.198.35 Author's email: [email protected]

Key Feature

• Search engines will automatically search for missing database search results from other search engines and automatically update them.
• Directly related to Youdao Dictionary, it is more convenient to search words
• There are no ads for the search, and the interface is simpler
• No trace search will not generate big data to kill mature
• Desimilation of blog pages such as csdn has been achieved, which has improved search efficiency (coming)
• The plug-in system will be launched soon
• Support real-time online feedback on bugs and suggestions
• Distributed crawler + self-implemented temporary database (CubeQL)

Running screenshot

CubeQL

A temporary storage structure for blush search, implementing redis-like functions, and can also filter crawler URLs through the Bloom filter module, and use it as deduplication

Speed ​​optimization is expected using vlang

0.1.1 Update Stage

1. Realize dynamic sorting of web pages for keywords in the database
2. Change the backend to flask, and the others still use fastapi

0.1.2 Update

Optimize the problem of 404 still inclusion in statuscode crawling by distributed crawlers, which increases search results web page sorting and weight increase (redirection)

0.1.3 Update

• Crawling summary of other search engines
• Fixed some bugs
• Implement the implementation of the bloomfilter Bloomfilter, allowing filtering to be directly separated from mysql. Next, it is necessary to implement the Mysql library to store data to reduce the burden.
• And implement a timer
• Selenium is used for crawlers from other search engines

0.1.4 Updated 2020/01/30

• Added Crisp API to facilitate users to submit comments
• Implemented the transfer of postgresql (from mysql)
• Added result crawler for bing (not implemented)
• Solved the previous optimization of the search associative word mechanism, and now the request will be sent as long as a new character is entered.
• Fixed known bugs for crawlers and cubeql

0.2 Updated 2020/02/27

• A bug was found for the search keywords for secondary word participle
• I discovered a bug in the data type of postgresql when crawling
• A new Bing crawler for keywords has been added. Now the keyword crawler that cannot be found will automatically go to Bing search to get it.
• Modified some logic problems of crawler and search main program

0.2.1 Updated 2021/08/05

• Added blister trend (/trend)
• Modified the rules for crawler processing queues in cubeql
• Fix known bugs

0.3.1 Updated 2021/09/29

• The extension system is released
• Added a new plugin: huyaoiBlog
• It is expected that a large part of the update will be related to plug-ins in the future

0.3.6 Updated 2021/12/15

• Many directory structures have been optimized
• Fixed many bugs
• And added the function of the word daily
• It seems that the performance has improved a little

User-agent

Mozilla/5.0 (compatible; VerdantSpider/1.0)

To-do

1. Add some necessary search engine weight dynamic update function (done)

2. Implement fuzzy search of Chinese->Pinyin (need to create a new mapping table)

3. Implement split database storage

4. Convert to postgresql(done)

5. Statistics the details of each search for easy summary (done)

6. Add distributed locks implemented by cubeql

7. Implement search words to save in the cloud, and the server does not require multiple crawlings

8. Implement the audio after clicking the volume icon

9. Implement vlang instead of most python functions and optimize performance

10. Realize daily search hot spots

11. Implement intelligent summary of various search engines

12. Use containers to manage the environment to realize one-click deployment and one-click operation

13. Life cycles for each single index of the search, and the weight will be reduced after a while (new programs are required to maintain) (done)

14. Implement simhash and implement deduplication of blog websites such as csdn

15. Supports diversified search and more user-friendly filters

Postgresql reference version is 11.10

Environment

python >= 3.6

flask

fastapi==0.54.1

psycopg2

starlette

requests_html

jieba

demjson

bloomfilter_live

gevent

Problems encountered when migrating postgresql

1. Postgresql only supports single quotes to include strings
2. Username is different from mysql
3. Postgresql has a pattern, but you don't need to pay attention to SQL modification, because it will automatically point to the public mode.

---

How to use

Directory structure

├─.vscode
├─docs
├─resources
│ ├─config
│ ├─CubeQL
│ └─__pycache__
│ ├─lib
│ ├─Spider
│ └─__pycache __ │ ├─static
│ │ ├─css
│ │ ├─img
│ └─music
│ ├─templates
│ └─__pycache __
└─sql

docs: Regularly store documents

resources: source code storage directory

config: The alternate directory for configuration file json (really called in the root directory, and the optimization of the directory structure has not been completed yet

CubeQL: a directory that stores CubeQL-related source code

Spider: directory for storing crawlers

lib: store requirements

static: store static files

templates: store template files

sql: store alternate table creation sql files

backend.py - Search engine backend file

config.json - configuration file, only the json file in the root directory where the code is running takes effect

Running steps

Maybe there are some libraries in the environment that are not mentioned, but anyway, what are there to install

Modify the database account password and ip of config.json

Make sure postgresql is open

Run in the CubeQL directory

uvicorn CubeQL:app --port 1278

Run in resources directory

python backend.py

Run it in the spider directory (you can open the green search if you don’t run it)

python CDS-Distributed.py

Green Trends

By accessing /trend, you can count the search frequency of all keywords

Plug-in system

Writing a plug-in system is quite simple. The plug-in system essentially has a front-end framework below the search results. You can register the plug-in through the extensions.json in the root directory (which will be placed in the config folder later).

Then create a new index.html and package.json in the extensions folder. For json, you can refer to huyaoiBlog

package.json