brewval

تقييم مطالبات تطبيقات LLM

في عصر العديد من مزودي نماذج النماذج واللغة الكبيرة التي تختلف في القدرات والسرعة والتكاليف حيث تكون هناك حاجة لتقييم المطالبات على مختلف مقدمي الخدمات والنماذج لاختيار المزيج الأنسب للمهمة المعطاة.

مثال

 from typing import Dict
from brewval . model import Prompt , Label
from brewval . eval import Evaluator

from langchain . llms import OpenAI , BaseLLM

prompt = Prompt ( """
Description: Feelings of disappointment, grief, hopelessness, disinterest, and dampened mood.
Emotion: sadness
Description: muscles become tense, your heart rate and respiration increase, and your mind becomes more alert, priming your body to either run from the danger or stand and fight
Emotion: fear
Description: {description}
Emotion: {result}""" )

labels = [
    Label ( 'fear' , { 'description' : 'heart rate and respiration increase' }),
    Label ( 'surprise' , { 'description' : 'quite brief and is characterized by a physiological startle response following something unexpected' }),
    Label ( 'anger' , { 'description' : 'Characterized by feelings of hostility, agitation, frustration, and antagonism towards others.' })
]

models : Dict [ str , BaseLLM ] = {
    'OpenAI[davinci-003]' : OpenAI ( model_name = 'text-davinci-003' ),
    'OpenAI[davinci-002]' : OpenAI ( model_name = 'text-davinci-002' ),
    'OpenAI[ada-001]' : OpenAI ( model_name = 'text-ada-001' )
}

evaluator = Evaluator ( models )

results = evaluator . evaluate ( prompt , labels )
for result in results :
    print ( f'Model { result . model_name } accuracy: { result . accuracy * 100 } %' )

المخرجات

 Model OpenAI[davinci-003] accuracy: 100.0%
Model OpenAI[davinci-002] accuracy: 33.3%
Model OpenAI[ada-001] accuracy: 0.0%

يثبت

تثبيت الشعر

 poetry install

 export OPENAI_API_KEY="your key"

تقييم

سطر الأوامر ، باستخدام البيانات من ملفات CSV:

 poetry run python3 -m brewval.cli -p examples/weather-umbrella/prompts.csv -l examples/weather-umbrella/labels.csv

دفتر Noteber Jupyter (مستندات/أمثلة/تقييم.

 poetry run jupyter notebook