Catch promptfoo Config Mistakes That validate Lets Through

An eval that fails loudly is the easy case. The expensive one passes. A test named “Digital goods are not refunded” goes green on every run because the customer message it was meant to send never reached the prompt, and the model answered a question nobody asked. Nothing in the output says so. The config parsed, the provider answered, and a not-icontains check had nothing to object to.

A refund-assistant config with four quiet faults

This is a short, realistic promptfoo config. It sets a shared refund policy in defaultTest and adds two test cases:

description: Refund assistant
prompts:
  - |
    You answer refund questions for an online store.
    Policy: {{policy}}
    Customer message: {{message}}
providers:
  - openai:gpt-5-mini
defaultTest:
  vars:
    policy: Refunds within 30 days of delivery. Digital goods are not refundable.
tests:
  - description: Late refund request is declined
    vars:
      message: I bought a jacket 45 days ago. Can I return it?
    assert:
      - type: contain
        value: 30 days
      - type: llm-rubric
        value: Declines politely and explains the 30-day window.
        threshold: 80
  - description: Digital goods are not refunded
    vars:
      customer_message: Please refund the e-book I bought yesterday.
    assert:
      - type: not-icontains
        value: refund has been issued

Every key in it is one promptfoo accepts, and every value has an accepted type. Checked with Ajv against the JSON schema published at the 0.123.1 tag, it validates. Choose Import YAML in the builder, paste it, and select Import config. The Findings panel lists one error, three warnings and one note:

error    tests[1] "Late refund request is declined" › assert[1]
         "contain" is not an assertion type in promptfoo 0.123.1. The schema accepts any
         string here, so `promptfoo validate` passes and the eval fails with "Unknown
         assertion type". Did you mean "contains"?
warning  tests[1] "Late refund request is declined" › assert[2]
         The threshold is 80, but the score it is compared with runs from 0 to 1.
         Nothing can reach it, so this always fails.
warning  tests[1] "Late refund request is declined" › assert[2]
         1 model-graded assertion has no grader. [...]
warning  tests[2] "Digital goods are not refunded"
         This test never sets {{message}}, which prompt 1 uses. [...]
info     vars.customer_message
         "customer_message" is set but no prompt or assertion reads it.

Why each one gets past the schema

The misspelt type. In promptfoo's own zod definitions, AssertionTypeSchema is a union that ends in an unrestricted z.custom(), and the published JSON schema ends the same union with an empty {}. Any string is a valid type. The check happens only when the assertion runs: the dispatcher in src/assertions/index.ts finds no handler for contain and throws Unknown assertion type: contain. The builder carries the 66 base types from the 0.123.1 schema, their not- forms, and the special types select-best, human and max-score, and suggests the nearest match for anything else.

The renamed variable. The second test sets customer_message, but the prompt reads {{message}}. promptfoo renders prompts with Nunjucks, and its getNunjucksEngine helper leaves throwOnUndefined at false, which the Nunjucks API documents as its default. An undefined value renders as an empty string. The model receives “Customer message:” followed by nothing, and a negative check such as not-icontains passes against almost any reply to a blank message. The builder reads the root name of every {{ }}, {% if %} and {% for … in … %} in the inline prompts, subtracts the vars in defaultTest and the test itself, and warns per test. It leaves out names promptfoo supplies, such as env and _conversation. In the form, a test that is missing variables shows an Add missing {{message}} button.

The threshold of 80. The llm-rubric grader is asked for a score from 0.0 to 1.0, and in 0.123.1 the grading code sets pass = pass && score >= threshold. A threshold of 80 reads naturally as a percentage and can never be met. The builder applies this range check only where the threshold really is a 0 to 1 score: llm-rubric, g-eval, similar, the context and relevance graders, ROUGE, BLEU and a test-level or assert-set threshold. For latency the threshold is milliseconds, for cost it is dollars and for levenshtein it is an edit count, and none of those are flagged for exceeding 1.

A key promptfoo does not know. This sample has none, but the commonest one is asserts: for assert:. The published JSON schema marks the config, each test and each assertion as closed objects, so an editor that honours the yaml-language-server comment underlines the typo. promptfoo's loader does not complain. TestCaseSchema is a plain z.object(), and zod strips keys it does not recognise instead of failing, so validate passes and the test runs with no assertions at all. The builder reports every unknown key as an error.

The missing grader. With no provider on the assertion, the test's options or defaultTest.options, llm-rubric picks its grader from whichever credentials are present. The docs list gpt-5 for an OpenAI key and claude-sonnet-4-5-20250929 for an Anthropic key, among others. A laptop with one key and a CI runner with another grade the same output with different models. This one is a warning rather than an error because --grader on the command line is a legitimate fix the builder cannot see.

Fix it in the form and export

1. Change the first assertion's type from contain to contains in its dropdown. The dropdown groups the common types as deterministic, model-graded and performance checks, and the not- checkbox beside it negates any of them.
2. Set the rubric threshold to 0.8.
3. Rename customer_message to message in the second test.
4. Type openai:gpt-5-mini into Grader for model-graded checks. It is written to defaultTest.options.provider, so every rubric in the file uses the same judge.
5. Check that the Findings panel is empty, then select Download to save promptfooconfig.yaml.

The changed parts of the exported file:

defaultTest:
  vars:
    policy: Refunds within 30 days of delivery. Digital goods are not refundable.
  options:
    provider: openai:gpt-5-mini
tests:
  - description: Late refund request is declined
    vars:
      message: I bought a jacket 45 days ago. Can I return it?
    assert:
      - type: contains
        value: 30 days
      - type: llm-rubric
        value: Declines politely and explains the 30-day window.
        threshold: 0.8
  - description: Digital goods are not refunded
    vars:
      message: Please refund the e-book I bought yesterday.

Run the file where your API key lives. promptfoo validate is still worth the second it takes. It rejects faults the builder also reports, such as a variable left as a bare message:, which YAML reads as null, or a file with neither providers nor targets, before any tokens are spent:

npx promptfoo@latest validate -c promptfooconfig.yaml
npx promptfoo@latest eval -c promptfooconfig.yaml

If the suite asserts latency, the run command shown under the YAML gains --no-cache. The latency handler throws on a cached response, which has no timing to measure. A latency or cost assertion without a threshold is reported as an error for the same reason: both handlers throw rather than guess.

Anything the form does not model survives an import. Transforms, rubricPrompt, evaluateOptions, env, test metadata and file:// test references are carried through and written back unchanged, so the builder can be used on an existing suite without flattening it.

CSV cells are read more strictly than they look

promptfoo loads a CSV of test cases with tests: file://tests.csv. Ordinary columns become vars, and each __expected column holds one assertion written as type: value. Add tests from CSV parses the rows the way src/csv.ts in 0.123.1 does and shows the result as editable test cases:

message,__expected1,__expected2
"Can I return shoes after 10 days?","contains: 30 days",latency
"Refund my gift card","contain: not refundable","latency(3000)"

Two of those four cells do not mean what they appear to. contain: not refundable does not match the parser's pattern of known types, so it falls through to the default and becomes an equals check on the entire string contain: not refundable. It fails on every output with no hint of why. A bare latency matches, but the parser supplies a default threshold of 0.75 to latency, cost and several other types when none is written in parentheses. That is a 0.75 ms budget. The builder warns on both cells; latency(3000) is the form that means three seconds.

What the builder does not check

It works on text in the config and nothing else. A prompt loaded with file://prompt.txt is not opened, so variables inside it are not compared with the tests, and the builder says so in a note rather than pretending the check passed. The same applies to vars loaded from files.

It does not run your javascript or python assertions, compile your JSON Schema against real outputs, or know which environment variables your CI runner exports. It cannot tell whether a model will wrap JSON in a Markdown fence, which makes is-json fail because that type parses the whole output; contains-json accepts embedded JSON.

The rules are pinned to promptfoo 0.123.1, the current release on 1 October 2026. npx promptfoo@latest will move on, and an assertion type added in a later version will be flagged as unknown here until the tool's list is updated. The version is printed in the page and in the YAML header so a mismatch is visible.

Questions about promptfoo configs

Should CI use promptfoo@latest? For a gate that decides merges, pin the version you validated against, for example npx [email protected] eval, and upgrade on purpose. @latest is fine for a local run, but it changes the evaluator under a suite that has not changed.

What does weight: 0 do? promptfoo marks an assertion with weight 0 as passed whatever it finds, which is useful for recording a metric without gating on it. The builder notes it so it is not mistaken for a disabled check.

Where does a schema for is-json come from? Write it from the output contract, or start from a captured response in the JSON to Schema Generator and review which fields are really required, as described in Review Required Fields in a Generated JSON Schema. Paste it into the assertion's value box as JSON or YAML; the export writes it as a mapping.

Is the builder a YAML formatter? No. A YAML parser can say the file is well-formed, which is the check in Review YAML Conversion Before Changing a CI File, but not that asserts should be assert. For a generated eval target, the reviewing habits in Review AI-Generated Playwright Tests Before You Merge Them translate directly into assertions: a not-contains on waitForTimeout is one line.

Apply this now

Import the promptfoo config your team runs most often. Resolve every error, then read each missing-var warning against the test's description: those are the tests most likely to have been passing without testing anything. Pin a grader, save the file, and run validate followed by eval. Keep the Findings list from before the fix in the pull request, so a reviewer can see which green results were never real.

promptfoo and Nunjucks references

• promptfoo: Assertions and metrics: assertion properties, not- negation, weights, test thresholds, and weight 0 passing automatically.
• promptfoo: Deterministic metrics: contains-any lists, is-json with a schema, latency needing --no-cache, cost thresholds.
• promptfoo: LLM Rubric: the 0.0 to 1.0 score, default graders per API key, and overriding the grader.
• promptfoo: Test case configuration: CSV test files and the __expected column syntax.
• promptfoo: Command line: validate, --no-cache, --grader and -r.
• promptfoo 0.123.1 config JSON schema: the assertion type list, and the closed key sets for the config, tests and assertions.
• promptfoo 0.123.1 src/csv.ts: the __expected parser, its fallback to equals, and the 0.75 default threshold.
• promptfoo 0.123.1 src/assertions/index.ts: the “Unknown assertion type” error at run time.
• Nunjucks API: throwOnUndefined defaults to false.
• AutomationTester.in LLM Eval Test-Case Builder: the import, findings and export shown above.