denser retriever

An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.

Description

Denser Retriever combines multiple search technologies into a single platform. It utilizes gradient boosting ( xgboost) machine learning technique to combine:

• Keyword-based searches that focus on fetching precisely what the query mentions.
• Vector databases that are great for finding a wide range of potentially relevant answers.
• Machine Learning rerankers that fine-tune the results to ensure the most relevant answers top the list.

• Our experiments on MTEB datasets show that the combination of keyword search, vector search and a reranker via a xgboost model (denoted as ES+VS+RR_n) can significantly improve the vector search (VS) baseline.

• Check out Denser Retriever experiments using the Anthropic Contextual Retrieval dataset at here.

Features

The initial release of Denser Retriever provides the following features.

• Supporting heterogeneous retrievers such as keyword search, vector search, and ML model reranking
• Leveraging xgboost ML technique to effectively combine heterogeneous retrievers
• State-of-the-art accuracy on MTEB Retrieval benchmarking
• Demonstrating how to use Denser retriever to power an end-to-end applications such as chatbot and semantic search

? Installation

We recommend installing Python via Anaconda, as we have received feedback about issues with Numpy installation when using the installer from https://www.python.org/downloads/. We are working on providing a solution to this problem. To install Denser Retriever, you can run:

Pip

pip install git+https://github.com/denser-org/denser-retriever.git#main

Poetry

poetry add git+https://github.com/denser-org/denser-retriever.git#main

? Documentation

The official documentation is hosted on retriever.denser.ai. Click here to get started.

??‍ Development

You can start developing Denser Retriever on your local machine.

See DEVELOPMENT.md for more details.

? License

This project is licensed under the terms of the MIT license. See LICENSE for more details.

? Citation

@misc{denser-retriever,
  author = {denser-org},
  title = {An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {url{https://github.com/denser-org/denser-retriever}}
}