brewval

LLM 응용 프로그램의 프롬프트를 평가합니다

기능, 속도 및 비용이 다른 여러 대형 언어 모델 제공 업체 및 모델의 시대에는 다른 공급자 및 모델에 대한 프롬프트를 평가하여 주어진 작업에 가장 적합한 조합을 선택해야합니다.

예

 from typing import Dict
from brewval . model import Prompt , Label
from brewval . eval import Evaluator

from langchain . llms import OpenAI , BaseLLM

prompt = Prompt ( """
Description: Feelings of disappointment, grief, hopelessness, disinterest, and dampened mood.
Emotion: sadness
Description: muscles become tense, your heart rate and respiration increase, and your mind becomes more alert, priming your body to either run from the danger or stand and fight
Emotion: fear
Description: {description}
Emotion: {result}""" )

labels = [
    Label ( 'fear' , { 'description' : 'heart rate and respiration increase' }),
    Label ( 'surprise' , { 'description' : 'quite brief and is characterized by a physiological startle response following something unexpected' }),
    Label ( 'anger' , { 'description' : 'Characterized by feelings of hostility, agitation, frustration, and antagonism towards others.' })
]

models : Dict [ str , BaseLLM ] = {
    'OpenAI[davinci-003]' : OpenAI ( model_name = 'text-davinci-003' ),
    'OpenAI[davinci-002]' : OpenAI ( model_name = 'text-davinci-002' ),
    'OpenAI[ada-001]' : OpenAI ( model_name = 'text-ada-001' )
}

evaluator = Evaluator ( models )

results = evaluator . evaluate ( prompt , labels )
for result in results :
    print ( f'Model { result . model_name } accuracy: { result . accuracy * 100 } %' )

출력

 Model OpenAI[davinci-003] accuracy: 100.0%
Model OpenAI[davinci-002] accuracy: 33.3%
Model OpenAI[ada-001] accuracy: 0.0%

설정

시를 설치하십시오

 poetry install

 export OPENAI_API_KEY="your key"

평가

CSV 파일의 데이터를 사용하여 명령 줄 :

 poetry run python3 -m brewval.cli -p examples/weather-umbrella/prompts.csv -l examples/weather-umbrella/labels.csv

Jupyter 노트북 (문서/예제/평가 .ipynb) :

 poetry run jupyter notebook