brewval

Eingabeaufforderungen für LLM -Anwendungen bewerten

In der Zeit mehrerer großer Sprachmodellanbieter und Modelle, die sich in den Fähigkeiten, Geschwindigkeit und Kosten unterscheiden, müssen die Eingabeaufforderungen für verschiedene Anbieter und Modelle bewertet werden, um die am besten geeignete Kombination für eine bestimmte Aufgabe auszuwählen.

Beispiel

 from typing import Dict
from brewval . model import Prompt , Label
from brewval . eval import Evaluator

from langchain . llms import OpenAI , BaseLLM

prompt = Prompt ( """
Description: Feelings of disappointment, grief, hopelessness, disinterest, and dampened mood.
Emotion: sadness
Description: muscles become tense, your heart rate and respiration increase, and your mind becomes more alert, priming your body to either run from the danger or stand and fight
Emotion: fear
Description: {description}
Emotion: {result}""" )

labels = [
    Label ( 'fear' , { 'description' : 'heart rate and respiration increase' }),
    Label ( 'surprise' , { 'description' : 'quite brief and is characterized by a physiological startle response following something unexpected' }),
    Label ( 'anger' , { 'description' : 'Characterized by feelings of hostility, agitation, frustration, and antagonism towards others.' })
]

models : Dict [ str , BaseLLM ] = {
    'OpenAI[davinci-003]' : OpenAI ( model_name = 'text-davinci-003' ),
    'OpenAI[davinci-002]' : OpenAI ( model_name = 'text-davinci-002' ),
    'OpenAI[ada-001]' : OpenAI ( model_name = 'text-ada-001' )
}

evaluator = Evaluator ( models )

results = evaluator . evaluate ( prompt , labels )
for result in results :
    print ( f'Model { result . model_name } accuracy: { result . accuracy * 100 } %' )

Ausgänge

 Model OpenAI[davinci-003] accuracy: 100.0%
Model OpenAI[davinci-002] accuracy: 33.3%
Model OpenAI[ada-001] accuracy: 0.0%

Aufstellen

Gedichte installieren

 poetry install

 export OPENAI_API_KEY="your key"

Auswertung

Befehlszeile, verwendete Daten aus CSV -Dateien:

 poetry run python3 -m brewval.cli -p examples/weather-umbrella/prompts.csv -l examples/weather-umbrella/labels.csv

Jupyter Notebook (Dokumente/Beispiele/Evaluation.IPynb):

 poetry run jupyter notebook