Vector Search with FAISS

Vector Search using Embeddings, FAISS and Product Quantization

Overview

This project implements an efficient similarity search system for lecture content using embeddings, FAISS and Product Quantization with custom index & KMeans implementations. It allows you to find similar lectures based on textual content, enabling quick retrieval and recommendation of lectures.

Features

• Data Preprocessing: Load and preprocess lecture and query data (Generated by ChatGPT).
• Embeddings: Compute and normalize embeddings using a specified model.
• FAISS Indexing: Build and evaluate a FAISS index for efficient similarity search.
• Performance Evaluation: Compute recall and queries per second (QPS) metrics.
• Quantization: Implement Product Quantization (PQ) with custom index to reduce storage requirements.
• Visualization: Plot performance metrics for analysis.

Table of Contents

• Installation
• Data Preparation
• Configuration
• Usage
• Dependencies
• License

Installation

1. Clone the Repository

   git clone https://github.com/bariscamli/Vector-Search-with-FAISS.git
   cd Vector-Search-with-FAISS

2. Create a Virtual Environment (Optional but Recommended)

   python -m venv venv
   source venv/bin/activate  # On Windows use `venvScriptsactivate`

3. Create a Virtual Environment (Optional but Recommended)

   pip install -r requirements.txt

Data Preparation

• Lecture Data: Place your lecture texts in a file specified by LECTURE_FILE in config.py. Each line should contain one lecture.

• Query Data: Place your query texts in a file specified by QUERY_FILE in config.py. Each line should contain one query. Example format for lectures.txt:

  Introduction to Machine Learning
  Advanced Topics in Deep Learning
  Statistical Methods in Data Science
  ...
  

  Example format for queries.txt:

  Basics of Neural Networks
  Regression Analysis Techniques
  Clustering Algorithms Overview
  ...
  

Configuration

All configurations are managed through the config.py file. Key parameters include:

File Paths
- LECTURE_FILE: Path to the lecture data file.
- QUERY_FILE: Path to the query data file.
Embedding Model
- EMBEDDING_MODEL_NAME: Name or path of the embedding model to use.
- BATCH_SIZE: Batch size for computing embeddings.
FAISS Parameters
- FAISS_EFSEARCH_VALUES: List of efSearch values for performance evaluation.
Quantization Parameters
- PQ_M: Number of sub-vector quantizers.
- PQ_NBITS: Number of bits per sub-vector.
- KMEANS_MAX_ITER: Maximum iterations for k-means during PQ training.

Usage

Run the main script to execute the full pipeline:

python main.py

What Happens When You Run main.py

1. Data Loading and Preprocessing

   • Lectures and queries are loaded from the specified files.
   • Text data is preprocessed (e.g., tokenization, cleaning).

2. Embedding Computation

   • An embedding model is loaded as per EMBEDDING_MODEL_NAME.
   • Embeddings for lectures and queries are computed and normalized.

3. Baseline Computation

   • A baseline similarity matrix is computed using dot products.
   • The baseline is used for performance comparison.

4. FAISS Index Building and Evaluation

   • A FAISS index is built for the lecture embeddings.
   • The index is evaluated over different efSearch values.
   • Performance metrics (recall@1 and QPS) are computed.

5. Performance Visualization

   • A plot is generated showing the trade-off between recall and QPS.
   • The plot is displayed using Matplotlib.

6. Quantization

   • A custom PQ index (CustomIndexPQ) is created.
   • The index is trained and lectures are added to it.

7. Example Search

   • An example search is performed using the PQ index.
   • Results are logged, showing similar lectures to a given lecture.

Dependencies

• Python 3.7 or higher
• Required Python packages (installed via requirements.txt):
  • numpy
  • matplotlib
  • faiss (Install via pip install faiss-cpu or faiss-gpu if you have a GPU)
  • logging
  • Embedding model libraries (e.g., transformers if using Hugging Face models)

License

This project is licensed under the MIT License. See the LICENSE

file for details.