weaviate_cohere_vector_embedding

This project is aimed at exploring Weaviate database search, using Cohere to generate custom vectors. There are several searches we perform using the steps listed below:

• RAG single prompt
• Vector search

You will need the following:

• a Weaviate API Key and URL
• a cohere API Key

1. install virtual env, then: pip install -r requirements.txt
2. create a .env file and store your creds
3. test the db connection by running: python database/open_db_connection
4. create a embeddings for the dataset: python database/create_embeddings

• this will create a new directory inside your root directory called 'scratch.'
• it will then create a .csv file that contains all of the vectors generated by Cohere

5. create a new collection in your Weaviate instance: python database/create_collection

• this will create a new collection called "MovieCustomVector"

6. import the data: python database/import_data
7. query the data: python -m queries.{the_file_you_wish_to_query}
8. to terminate the db connection for any reason: python database/close_db_connection

More info on search methods

RAG search, aka 'Generative', consists of 2 steps:

1. (Weaviate) Run a search query in Weaviate to find relevant objects.
2. (Cohere) Use a Cohere Large Language Model to generate a response based on the results (from the previous step) and the provided prompt or task.

Testing REST endpoints

• paste your WCD REST Endpoint URL into the browser, and append it with /v1
• this will show you a list of RESTful endpoints you can access by configuring Postman.
• Weaviate uses a bearer token, and you must paste in your WCD_API_KEY for authorization