VDTuner
1.0.0
VDTuner在配置CentOS 7.9.2009（Linux 5.5.5）和Python 3.11的服務器上進行了測試。評估的矢量數據庫管理系統和Benchamark是Mivlus（2.3.1版）和矢量DB基準。
請參閱http://arxiv.org/abs/2404.10413的預印本
docker-compose.yml (Milvus startup file), milvus.yaml (Milvus configuration file for utilizing) and milvus.yaml.backup (Milvus configuration file for copying and modifying) in your benchmark path vector-db-benchmark-master/engine/servers/milvus-single-node .run_engine.sh複製到vector-db-benchmark-master/run_engine.sh 。vector-db-benchmark-master並運行： sudo ./run_engine.sh "" "" random-100 。 vector-db-benchmark-master/datasets/glove-100-angular/glove-100-angular.hdf5 。 將文件vector-db-benchmark-master/experiments/configurations/milvus-single-node.json修改為虛假索引配置如下。參數parallel可以根據您的服務器規格進行修改。
[
{
"name" : " milvus-p10 " ,
"engine" : " milvus " ,
"connection_params" : {},
"collection_params" : {},
"search_params" : [
{
"parallel" : 10 ,
"params" : {}
}
],
"upload_params" : {
"parallel" : 10 ,
"index_type" : " AUTOINDEX " ,
"index_params" : {}
}
}
]指定您的數據集和超時限制。在文件auto-configure/vdtuner/utils.py （第117行）中，假設我們測試數據集手套，每個工作負載重播最多15分鐘：
result = sp . run ( f'sudo timeout 900 { RUN_ENGINE_PATH } "" "" glove-100-angular' , shell = True , stdout = sp . PIPE )要運行vdtuner，您需要指定調整參數和基準路徑的配置文件。這是一個例子。
在文件auto-configure/configure.py中，第4-9行：
with open ( '/home/ytn/milvusTuning/auto-configure/index_param.json' , 'r' ) as f :
INDEX_PARAM_DICT = json . load ( f )
CONF_PATH = r'/home/ytn/milvusTuning/vector-db-benchmark-master/experiments/configurations/milvus-single-node.json'
ORIGIN_PATH = r'/home/ytn/milvusTuning/vector-db-benchmark-master/engine/servers/milvus-single-node/milvus.yaml.backup'
ADJUST_PATH = r'/home/ytn/milvusTuning/vector-db-benchmark-master/engine/servers/milvus-single-node/milvus.yaml'在文件auto-configure/vdtuner/utils.py中，第13-14行：
KNOB_PATH = r'/home/ytn/milvusTuning/auto-configure/whole_param.json'
RUN_ENGINE_PATH = r'/home/ytn/milvusTuning/vector-db-benchmark-master/run_engine.sh' auto-configure/vdtuner/並運行： python3.11 main_tuner.pymain_tuner.py中按所需的迭代數量更改。record.log和pobo_record.log 。這是record.log中的示例輸出： [1] 125 {index_type: FLAT, nlist: 128, nprobe: 10, m: 10, nbits: 8, M: 32, efConstruction: 256, ef: 500, reorder_k: 500} {dataCoord*segment*maxSize: 512, dataCoord*segment*sealProportion: 0.23, queryCoord*autoHandoff: True, queryCoord*autoBalance: True, common*gracefulTime: 5000, dataNode*segment*insertBufSize: 16777216, rootCoord*minSegmentSizeToEnableIndex: 1024} 230.5802223315391 0.9999830000000002 125
[2] 214 {index_type: IVF_FLAT, nlist: 128, nprobe: 10, m: 10, nbits: 8, M: 32, efConstruction: 256, ef: 500, reorder_k: 500} {dataCoord*segment*maxSize: 512, dataCoord*segment*sealProportion: 0.23, queryCoord*autoHandoff: True, queryCoord*autoBalance: True, common*gracefulTime: 5000, dataNode*segment*insertBufSize: 16777216, rootCoord*minSegmentSizeToEnableIndex: 1024} 1086.9213657571365 0.8496440000000001 88
[3] 302 {index_type: IVF_SQ8, nlist: 128, nprobe: 10, m: 10, nbits: 8, M: 32, efConstruction: 256, ef: 500, reorder_k: 500} {dataCoord*segment*maxSize: 512, dataCoord*segment*sealProportion: 0.23, queryCoord*autoHandoff: True, queryCoord*autoBalance: True, common*gracefulTime: 5000, dataNode*segment*insertBufSize: 16777216, rootCoord*minSegmentSizeToEnableIndex: 1024} 908.6127610863159 0.8461550000000001 88
...
如果您在科學文章中使用vdtuner，請引用我們的ICDE 2024論文：
@inproceedings{yang2024vdtuner,
title={VDTuner: Automated Performance Tuning for Vector Data Management Systems},
author={Yang, Tiannuo and Hu, Wen and Peng, Wangqi and Li, Yusen and Li, Jianguo and Wang, Gang and Liu, Xiaoguang},
booktitle={2024 IEEE 40th International Conference on Data Engineering (ICDE)},
year={2024}
}
tiannuo yang [email protected]
wangqi peng [email protected]
- 來自實驗室NBJL和螞蟻組
