LLM Eval Test-Case Builder
Write promptfoo test cases without hand-editing YAML. Add prompts with {{variables}}, test cases, and assertions such as equals, contains-any, regex, is-json with a schema, javascript, llm-rubric, similar, latency and cost, with not- negation, weights, thresholds and metrics. Import an existing promptfooconfig.yaml or a test CSV to edit it. The builder checks the result against the promptfoo 0.123.1 config schema and flags what validation lets through: unknown assertion types, variables a test never sets, rubrics with no grader, and thresholds outside 0 to 1. It runs in your browser and never calls a model.
Eval Case Builder
Test cases to promptfooconfig.yaml
Runs entirely in your browser. Nothing you type is sent anywhere, and this tool never calls a model: it writes the config, and you run the eval on your machine with npx promptfoo@latest eval. Checks follow the promptfoo 0.123.1 config schema.
Variables the prompts read: {{tier}}{{ticket}}
Written to defaultTest.options.provider. Leave empty and promptfoo picks a grader from whichever API key it finds.
The whole output parses as JSON; an optional JSON Schema is checked too.
The provider call took at most this many milliseconds. Needs --no-cache.
An expression or function body with `output` in scope; return a boolean, a score, or a GradingResult.
Output contains the value, ignoring case.
Like contains-any, ignoring case.
A grader model scores the output against this rubric.
Output matches a JavaScript regular expression.
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
# Checked against promptfoo 0.123.1 config rules by the AutomationTester.in LLM Eval Test-Case Builder.
# Run: npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache
description: Support ticket triage
prompts:
- |-
You triage support tickets for an online store.
Reply with JSON only: {"category": "billing" | "shipping" | "account", "priority": "low" | "high", "reply": string}.
Customer tier: {{tier}}
Ticket: {{ticket}}
providers:
- openai:gpt-5-mini
defaultTest:
assert:
- type: is-json
value:
type: object
required:
- category
- priority
- reply
properties:
category:
enum:
- billing
- shipping
- account
priority:
enum:
- low
- high
reply:
type: string
- type: latency
threshold: 8000
options:
provider: openai:gpt-5-mini
tests:
- description: Double charge is billing, high priority
vars:
tier: gold
ticket: I was charged twice for order 1182. Please refund one of them.
assert:
- type: javascript
value: JSON.parse(output).category === 'billing'
- type: icontains
value: '1182'
- description: Late parcel gets an apology, not a refund promise
vars:
tier: standard
ticket: My parcel was due Monday and still has not arrived.
assert:
- type: not-icontains-any
value:
- refund issued
- we have refunded
- type: llm-rubric
value: The reply apologises for the delay and does not promise a refund.
threshold: 0.8
- description: Password reset is account
vars:
ticket: I cannot log in and the reset email never arrives.
assert:
- type: regex
value: '"category":\s*"account"'
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cachetests[3] "Password reset is account"
This test never sets {{tier}}, which prompt 1 uses. promptfoo renders an undefined variable as an empty string, so the model is asked the question with a blank where "tier" should be, and nothing errors.
defaultTest › assert[2]
latency only works on fresh calls. Run the eval with --no-cache; a cached response has no latency and the assertion errors.
Frequently Asked Questions(click to expand)
Does this run my eval?
No. It writes promptfooconfig.yaml and checks it. Save the file, set the API key your provider needs, and run npx promptfoo@latest eval -c promptfooconfig.yaml. npx promptfoo@latest validate -c promptfooconfig.yaml runs promptfoo's own schema check first.
Why flag an assertion type that promptfoo validate accepts?
The published schema lets type be any string, so a typo such as contain passes validation. The eval then stops with "Unknown assertion type". The builder compares every type against the promptfoo 0.123.1 list.
is-json or contains-json?
is-json parses the whole output, so a reply wrapped in a Markdown code fence fails it. contains-json accepts JSON embedded in other text. Both take an optional JSON Schema as the value.
What happens to a variable a test forgets?
promptfoo renders prompts with Nunjucks and leaves undefined variables blank. The prompt still goes to the model, just with a gap, so the builder warns per test instead.
Related
Format or convert the finished file in the YAML Formatter, generate a schema for an is-json check with the JSON to Schema Generator, or prepare CSV test data in the CSV ↔ JSON Converter.