VecTextSearch Download - VecTextSearch Source code download

VecTextSearch

Other source code

v0.1.3

Download

VecTextSearch

VecTextSearch is a project that uses the OpenAI language model to generate text vectors and perform efficient searches in the Weaviate database. It allows users to store text data in a Weaviate database and quickly search and retrieve relevant text based on text similarity. The project is written in Golang and provides a simple REST API for client calls

Simplified Chinese | English

Project Introduction

VecTextSearch is a project that uses the OpenAI language model to generate text vectors and perform efficient searches in a Weaviate database. It allows users to store text data in a Weaviate database and quickly search and retrieve relevant text based on text similarity. The project is written in Golang and provides a simple REST API for client calls.

Chat history

Chat History 1 - Create a Project

Chat History 2 - Modify Dockerfile and Makefile

Chat Record 3 - Simplify the return results of vector search and modify the data structure

Chat Record 4 - Refactoring the project structure

Chat Record 5 - Download ChatGPT Chat Dialogue directly as Markdown file

Chat History 6 - Add cross-domain support to fix the error of make run command

Chat History 7 - Fix Document Errors

Chat history 8 - Configure whether the content can be repeated when adding text

Chat history 9 - weaviate's className can be configured

Effect

postman

web

flutter

Project background

In many practical applications, quick searches are required based on text similarity. For example, given an article, you can find other articles that are similar to their content. Traditional keyword-based search methods may not accurately capture similarities between texts. VecTextSearch uses OpenAI's powerful language model to convert text into vector representations and then uses the Weaviate database for efficient similar vector search.

Uses and usage scenarios

VecTextSearch can be applied to the following scenarios:

Find relevant content for articles, blogs, papers, etc.
Implement an intelligent question-and-answer system to quickly match relevant questions and answers based on user questions.
Build a recommendation system and recommend similar articles to users based on their reading history.
Detect duplicate or plagiarized content.

TODO List

Develop a demo application : Create a demo application that visualizes VecTextSearch functionality and use cases.
Add a data management interface : Provides a data management interface for projects to facilitate users to manage text data stored in the Weaviate database.
Develop a user-friendly front-end interface : simplify the use of VecTextSearch and provide users with a better experience.
Provide detailed documentation : Write detailed documentation including API references, usage examples, and tutorials.
More configuration options are available : Allows users to adjust the performance and functionality of VecTextSearch according to their needs.
Add unit tests and integration tests : Ensure code quality and stability.
Pay attention to the updates of OpenAI language model : Continue to pay attention to the updates and improvements of OpenAI language model, and timely apply the latest technologies to VecTextSearch.
Development plug-in or extension system : allows users to customize the functions of VecTextSearch according to their needs.

Interface introduction

VecTextSearch provides two REST API interfaces:

Add text

URL: /add-text
Method: POST
Content-Type: application/json
Request Payload:

{
  "name" : "文章名称" ,
  "content" : "文章内容"
}

Response: After successfully adding text, a JSON object containing the text ID will be returned.

{
  "id" : "文章唯一标识符"
}

Search for similar texts

URL: /search-similar-texts
Method: POST
Content-Type: application/json
Request Payload:

{
  "content" : "查询内容"
}

Response: After the search is successful, a JSON object containing similar text information will be returned.

[
  {
    "name" : "文章名称" ,
    "content" : "文章内容" ,
    "distance" : 浮点数（与查询内容的距离）,
    "certainty" : 浮点数（与查询内容的相似度）
  },
  ...
]

Makefile function description

make init : Creates a .env file template for configuring environment variables.
make build : build Docker image.
make push : Push the Docker image to the Docker Hub.
make run : Run the application locally.

Start the Weaviate vector database

docker run -d 
  --name weaviate 
  -p 8888:8080 
  --restart on-failure:0 
  -e QUERY_DEFAULTS_LIMIT=25 
  -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true 
  -e PERSISTENCE_DATA_PATH= ' /var/lib/weaviate ' 
  -e DEFAULT_VECTORIZER_MODULE= ' none ' 
  -e ENABLE_MODULES= ' ' 
  -e AUTOSCHEMA_ENABLED=true 
  -e CLUSTER_HOSTNAME= ' node1 ' 
  semitechnologies/weaviate:1.18.1 
  --host 0.0.0.0 
  --port 8080 
  --scheme http

ChatGPT to Markdown Chrome Plugin

ChatGPT to Markdown is a Chrome plug-in developed by ChatGPT, designed to help users easily download ChatGPT and OpenAI conversation records as Markdown files. The generated Markdown file will contain the entire conversation and make a clear distinction between the user and the assistant. This plug-in is convenient for users to organize and view chat history and improve work efficiency.

Main functions:

Add a "Download Markdown" button to the ChatGPT dialog page
Convert the entire conversation record to Markdown format
Automatically generate chat history paragraphs with titles "Neo" (user) and "ChatGPT" (assistant)

For detailed instructions and usage methods, please refer to the ChatGPT to Markdown plug-in file.

Development and Contribution

If you want to contribute to VecTextSearch or secondary development of your project, you can follow these steps:

Cloning this repository to local:

git clone https://github.com/szpnygo/VecTextSearch.git

Enter the project directory and install the relevant dependencies:

 cd VecTextSearch
go get -u

Fill in the correct OpenAI API key in the config.yml file.
Run the project:

go run main.go

If you have problems using VecTextSearch or have new ideas and suggestions, please submit an Issue or Pull Request. We really appreciate your contribution and support!