markdown file query下载 - markdown file query源代码下载

markdown file query

其他源码

1.0.0

下载

该项目目前在英文文档中最有效。

关于这个项目

这个项目

利用Pinecone Vector数据库（VDB）和OpenAI（Vector）嵌入模型将文本变成向量。
与任何.md文件一起使用，因此它可以与Intion＆Intidian完美搭配（尽管对于概念，您必须先手动将其导出到.md ）
是作者的Feynman技术实践。
可能是Privategpt和Llama_index的弱重复，如果您想要精美的文档查询程序，则应使用llama_index而不是此玩具。

该计划的演练

使用langchain.textsplitter将目标目录中的每个标记文件切成许多小块
每个chunck都通过Openai的嵌入模型（ langchain.embeddings.OpenAIEmbeddings ）变成矢量
然后将向量上传到Pinecone矢量数据库。
查询还使用矢量嵌入模型转换为向量，并将其上传到Pinecone。
为了检索搜索结果，我们使用Pinecone（通过余弦相似性）将查询向量与向量数据库进行比较。
最接近的3个结果将被检索并送入GPT-3以及问题，而GPT-3将以自然语言产生答案。

托多

添加一个--help选项
部署到简化

入门

先决条件

准备Pinecone和OpenAI API密钥：
- 可以在此处获得Pinecone API键。
- 可以在此处获得OpenAI API密钥。

将松果和OpenAI API键导出到系统环境

 export PINECONE_API_KEY= " your_pinecone_api_key "
export OPENAI_API_KEY= " your_openai_api_key "

现在使用Python

 import os
os . environ [ "PINECONE_API_KEY" ]
os . environ [ "OPENAI_API_KEY" ]

要检查是否将它们导出到系统环境中，如果KeyError ，请在完成后重新启动终端（如果您使用的是IDE）。

安装

克隆此存储库到您当地的机器

git clone https://github.com/madeyexz/markdown-file-query.git

安装依赖项
```
pip install pinecone langchain tqdm
```

用法

准备Markdown文件并将其放入FOLDER （或您喜欢的任何名称，但您必须相应地更改代码）。请注意，这应该与main.py处于同一目录。
如果这是您第一次查询某个文档，请运行main.py程序
```
python3 main.py " PATH_OF_FOLDER " " QUESTION "
```
查询结果和用于生成答案的参考GPT将分别保存在answer.txt和contents.txt中。
如果您想再次查询相同的文档，请运行query_only.py以避免重新安装文档。
```
python3 query_only.py " QUESTION "
```

例子

我有一个名为markdown_database的文件夹，其中包含一堆.md文件，我想用“什么奇怪情况”查询此数据库

❯ python3 main.py " markdown_database " " what's the strange situation "

 initiating pinecone index...
digesting docs...
uploading datas to pinecone...
92%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████          | 60/65 [00:29<00:02,  1.87it/s]
let's wait for 60 seconds to avoid RateLimitError... (since im not a paid user))
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [01:00<00:00,  1.00s/it]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 65/65 [01:32<00:00,  1.42s/it]
querying pinecone...
querying gpt...
writing results to answer.txt and contents.txt
done! the answer to 'what's the strange situation' is: '
The Strange Situation is a standardized procedure devised by Mary Ainsworth in the 1970s to observe attachment security in children within the context of caregiver relationships. It applies to infants between the age of nine and 18 months and involves a series of eight episodes lasting approximately 3 minutes each, whereby a mother, child and stranger are introduced, separated and reunited. The procedure is used to observe the quality of a young child’s attachment to his or her mother, and can also be applied to other attachment figures, such as God, through the use of Emotionally Focused Therapy (EFT) and religious beliefs, such as the saying “there are no atheists in foxholes”.'

如果我想再次查询相同的数据库，我可以使用query_only.py来避免重新安装文档。

❯ python3 query_only.py " Who is Mary Ainsworth? "

 connecting to pinecone index...
getting docs
querying pinecone...
querying gpt...
done! the answer to 'Who is Mary Ainsworth?' is: '
Mary Ainsworth was a developmental psychologist who devised the Strange Situation in the 1970s to observe attachment security in children within the context of caregiver relationships. The Strange Situation involves a series of eight episodes lasting approximately 3 minutes each, whereby a mother, child and stranger are introduced, separated and reunited. Ainsworth is also known for her observation that if you want to see the quality of a young child’s attachment to his or her mother, watch what the child does, not when Mother leaves, but when she returns. She is also known for her research on anxious babies and their inability to use their mothers as a secure base.'

已知限制

如果您使用Pinecone，那么每当您想查询新文档（即创建新数据库）时，您可能应该创建一个新的Pinecone索引（因为您不希望从旧文档中的答案）或删除旧索引。这是因为Pinecone不支持更新索引（尚未）。
删除旧索引：
```
python3 delete_pinecone_index.py NAME_OF_INDEX
```