# Catch promptfoo Config Mistakes That validate Lets Through > Import a promptfooconfig.yaml into a browser-only builder and see what the 0.123.1 schema lets through: unknown assertion types, prompts that render with a blank variable, unpinned graders, 0-to-1 thresholds written as percentages, and CSV cells that silently become equals checks. - Author: [Shashank Rawlani](https://shashank.rawlani.com) - Published: 2026-10-30T05:30:00.000Z - Updated: 2026-10-30T05:30:00.000Z - Category: Tools & Reviews - Canonical source: https://automationtester.in/blog/tools-reviews/catch-promptfoo-config-mistakes-before-eval - Tags: promptfoo, llm-evals, llm-testing, ai-testing, yaml **Quick answer:** Paste your `promptfooconfig.yaml` into the [LLM Eval Test-Case Builder](https://automationtester.in/tools/llm-eval-case-builder), or build one in its form. It checks the file against the promptfoo 0.123.1 config schema and then reports four faults that schema accepts: an assertion type promptfoo does not have, a `{{variable}}` a test never sets, a model-graded check with no grader pinned, and a score threshold outside 0 to 1. Fix them in the form, download the file, and run `npx promptfoo@latest eval` yourself. The tool never calls a model. An eval that fails loudly is the easy case. The expensive one passes. A test named “Digital goods are not refunded” goes green on every run because the customer message it was meant to send never reached the prompt, and the model answered a question nobody asked. Nothing in the output says so. The config parsed, the provider answered, and a `not-icontains` check had nothing to object to. ## A refund-assistant config with four quiet faults This is a short, realistic promptfoo config. It sets a shared refund policy in `defaultTest` and adds two test cases: ``` description: Refund assistant prompts: - | You answer refund questions for an online store. Policy: {{policy}} Customer message: {{message}} providers: - openai:gpt-5-mini defaultTest: vars: policy: Refunds within 30 days of delivery. Digital goods are not refundable. tests: - description: Late refund request is declined vars: message: I bought a jacket 45 days ago. Can I return it? assert: - type: contain value: 30 days - type: llm-rubric value: Declines politely and explains the 30-day window. threshold: 80 - description: Digital goods are not refunded vars: customer_message: Please refund the e-book I bought yesterday. assert: - type: not-icontains value: refund has been issued ``` [Try this in LLM Eval Test-Case Builder](https://automationtester.in/tools/llm-eval-case-builder#p=eyJ2IjoxLCJpbnB1dCI6ImRlc2NyaXB0aW9uOiBSZWZ1bmQgYXNzaXN0YW50XG5wcm9tcHRzOlxuICAtIHxcbiAgICBZb3UgYW5zd2VyIHJlZnVuZCBxdWVzdGlvbnMgZm9yIGFuIG9ubGluZSBzdG9yZS5cbiAgICBQb2xpY3k6IHt7cG9saWN5fX1cbiAgICBDdXN0b21lciBtZXNzYWdlOiB7e21lc3NhZ2V9fVxucHJvdmlkZXJzOlxuICAtIG9wZW5haTpncHQtNS1taW5pXG5kZWZhdWx0VGVzdDpcbiAgdmFyczpcbiAgICBwb2xpY3k6IFJlZnVuZHMgd2l0aGluIDMwIGRheXMgb2YgZGVsaXZlcnkuIERpZ2l0YWwgZ29vZHMgYXJlIG5vdCByZWZ1bmRhYmxlLlxudGVzdHM6XG4gIC0gZGVzY3JpcHRpb246IExhdGUgcmVmdW5kIHJlcXVlc3QgaXMgZGVjbGluZWRcbiAgICB2YXJzOlxuICAgICAgbWVzc2FnZTogSSBib3VnaHQgYSBqYWNrZXQgNDUgZGF5cyBhZ28uIENhbiBJIHJldHVybiBpdD9cbiAgICBhc3NlcnQ6XG4gICAgICAtIHR5cGU6IGNvbnRhaW5cbiAgICAgICAgdmFsdWU6IDMwIGRheXNcbiAgICAgIC0gdHlwZTogbGxtLXJ1YnJpY1xuICAgICAgICB2YWx1ZTogRGVjbGluZXMgcG9saXRlbHkgYW5kIGV4cGxhaW5zIHRoZSAzMC1kYXkgd2luZG93LlxuICAgICAgICB0aHJlc2hvbGQ6IDgwXG4gIC0gZGVzY3JpcHRpb246IERpZ2l0YWwgZ29vZHMgYXJlIG5vdCByZWZ1bmRlZFxuICAgIHZhcnM6XG4gICAgICBjdXN0b21lcl9tZXNzYWdlOiBQbGVhc2UgcmVmdW5kIHRoZSBlLWJvb2sgSSBib3VnaHQgeWVzdGVyZGF5LlxuICAgIGFzc2VydDpcbiAgICAgIC0gdHlwZTogbm90LWljb250YWluc1xuICAgICAgICB2YWx1ZTogcmVmdW5kIGhhcyBiZWVuIGlzc3VlZCJ9) Every key in it is one promptfoo accepts, and every value has an accepted type. Checked with Ajv against the JSON schema published at the 0.123.1 tag, it validates. Choose **Import YAML** in the builder, paste it, and select **Import config**. The Findings panel lists one error, three warnings and one note: ``` error tests[1] "Late refund request is declined" › assert[1] "contain" is not an assertion type in promptfoo 0.123.1. The schema accepts any string here, so `promptfoo validate` passes and the eval fails with "Unknown assertion type". Did you mean "contains"? warning tests[1] "Late refund request is declined" › assert[2] The threshold is 80, but the score it is compared with runs from 0 to 1. Nothing can reach it, so this always fails. warning tests[1] "Late refund request is declined" › assert[2] 1 model-graded assertion has no grader. [...] warning tests[2] "Digital goods are not refunded" This test never sets {{message}}, which prompt 1 uses. [...] info vars.customer_message "customer_message" is set but no prompt or assertion reads it. ``` ## Why each one gets past the schema **The misspelt type.** In promptfoo's own zod definitions, `AssertionTypeSchema` is a union that ends in an unrestricted `z.custom()`, and the published JSON schema ends the same union with an empty `{}`. Any string is a valid `type`. The check happens only when the assertion runs: the dispatcher in `src/assertions/index.ts` finds no handler for `contain` and throws `Unknown assertion type: contain`. The builder carries the 66 base types from the 0.123.1 schema, their `not-` forms, and the special types `select-best`, `human` and `max-score`, and suggests the nearest match for anything else. **The renamed variable.** The second test sets `customer_message`, but the prompt reads `{{message}}`. promptfoo renders prompts with Nunjucks, and its `getNunjucksEngine` helper leaves `throwOnUndefined` at `false`, which the Nunjucks API documents as its default. An undefined value renders as an empty string. The model receives “Customer message:” followed by nothing, and a negative check such as `not-icontains` passes against almost any reply to a blank message. The builder reads the root name of every `{{ }}`, `{% if %}` and `{% for … in … %}` in the inline prompts, subtracts the vars in `defaultTest` and the test itself, and warns per test. It leaves out names promptfoo supplies, such as `env` and `_conversation`. In the form, a test that is missing variables shows an **Add missing {{message}}** button. **The threshold of 80.** The `llm-rubric` grader is asked for a score from 0.0 to 1.0, and in 0.123.1 the grading code sets `pass = pass && score >= threshold`. A threshold of 80 reads naturally as a percentage and can never be met. The builder applies this range check only where the threshold really is a 0 to 1 score: `llm-rubric`, `g-eval`, `similar`, the context and relevance graders, ROUGE, BLEU and a test-level or assert-set `threshold`. For `latency` the threshold is milliseconds, for `cost` it is dollars and for `levenshtein` it is an edit count, and none of those are flagged for exceeding 1. **A key promptfoo does not know.** This sample has none, but the commonest one is `asserts:` for `assert:`. The published JSON schema marks the config, each test and each assertion as closed objects, so an editor that honours the `yaml-language-server` comment underlines the typo. promptfoo's loader does not complain. `TestCaseSchema` is a plain `z.object()`, and zod strips keys it does not recognise instead of failing, so `validate` passes and the test runs with no assertions at all. The builder reports every unknown key as an error. **The missing grader.** With no provider on the assertion, the test's `options` or `defaultTest.options`, `llm-rubric` picks its grader from whichever credentials are present. The docs list `gpt-5` for an OpenAI key and `claude-sonnet-4-5-20250929` for an Anthropic key, among others. A laptop with one key and a CI runner with another grade the same output with different models. This one is a warning rather than an error because `--grader` on the command line is a legitimate fix the builder cannot see. ## Fix it in the form and export - Change the first assertion's type from `contain` to `contains` in its dropdown. The dropdown groups the common types as deterministic, model-graded and performance checks, and the `not-` checkbox beside it negates any of them. - Set the rubric threshold to `0.8`. - Rename `customer_message` to `message` in the second test. - Type `openai:gpt-5-mini` into **Grader for model-graded checks**. It is written to `defaultTest.options.provider`, so every rubric in the file uses the same judge. - Check that the Findings panel is empty, then select **Download** to save `promptfooconfig.yaml`. The changed parts of the exported file: ``` defaultTest: vars: policy: Refunds within 30 days of delivery. Digital goods are not refundable. options: provider: openai:gpt-5-mini tests: - description: Late refund request is declined vars: message: I bought a jacket 45 days ago. Can I return it? assert: - type: contains value: 30 days - type: llm-rubric value: Declines politely and explains the 30-day window. threshold: 0.8 - description: Digital goods are not refunded vars: message: Please refund the e-book I bought yesterday. ``` Run the file where your API key lives. `promptfoo validate` is still worth the second it takes. It rejects faults the builder also reports, such as a variable left as a bare `message:`, which YAML reads as null, or a file with neither `providers` nor `targets`, before any tokens are spent: ``` npx promptfoo@latest validate -c promptfooconfig.yaml npx promptfoo@latest eval -c promptfooconfig.yaml ``` If the suite asserts `latency`, the run command shown under the YAML gains `--no-cache`. The latency handler throws on a cached response, which has no timing to measure. A `latency` or `cost` assertion without a threshold is reported as an error for the same reason: both handlers throw rather than guess. Anything the form does not model survives an import. Transforms, `rubricPrompt`, `evaluateOptions`, `env`, test metadata and `file://` test references are carried through and written back unchanged, so the builder can be used on an existing suite without flattening it. ## CSV cells are read more strictly than they look promptfoo loads a CSV of test cases with `tests: file://tests.csv`. Ordinary columns become vars, and each `__expected` column holds one assertion written as `type: value`. **Add tests from CSV** parses the rows the way `src/csv.ts` in 0.123.1 does and shows the result as editable test cases: ``` message,__expected1,__expected2 "Can I return shoes after 10 days?","contains: 30 days",latency "Refund my gift card","contain: not refundable","latency(3000)" ``` Two of those four cells do not mean what they appear to. `contain: not refundable` does not match the parser's pattern of known types, so it falls through to the default and becomes an `equals` check on the entire string `contain: not refundable`. It fails on every output with no hint of why. A bare `latency` matches, but the parser supplies a default threshold of `0.75` to latency, cost and several other types when none is written in parentheses. That is a 0.75 ms budget. The builder warns on both cells; `latency(3000)` is the form that means three seconds. ## What the builder does not check It works on text in the config and nothing else. A prompt loaded with `file://prompt.txt` is not opened, so variables inside it are not compared with the tests, and the builder says so in a note rather than pretending the check passed. The same applies to vars loaded from files. It does not run your `javascript` or `python` assertions, compile your JSON Schema against real outputs, or know which environment variables your CI runner exports. It cannot tell whether a model will wrap JSON in a Markdown fence, which makes `is-json` fail because that type parses the whole output; `contains-json` accepts embedded JSON. The rules are pinned to promptfoo 0.123.1, the current release on 1 October 2026. `npx promptfoo@latest` will move on, and an assertion type added in a later version will be flagged as unknown here until the tool's list is updated. The version is printed in the page and in the YAML header so a mismatch is visible. ## Questions about promptfoo configs **Should CI use `promptfoo@latest`?** For a gate that decides merges, pin the version you validated against, for example `npx promptfoo@0.123.1 eval`, and upgrade on purpose. `@latest` is fine for a local run, but it changes the evaluator under a suite that has not changed. **What does `weight: 0` do?** promptfoo marks an assertion with weight 0 as passed whatever it finds, which is useful for recording a metric without gating on it. The builder notes it so it is not mistaken for a disabled check. **Where does a schema for `is-json` come from?** Write it from the output contract, or start from a captured response in the [JSON to Schema Generator](https://automationtester.in/tools/json-schema) and review which fields are really required, as described in [Review Required Fields in a Generated JSON Schema](https://automationtester.in/blog/tools-reviews/review-generated-json-schema-required-fields). Paste it into the assertion's value box as JSON or YAML; the export writes it as a mapping. **Is the builder a YAML formatter?** No. A YAML parser can say the file is well-formed, which is the check in [Review YAML Conversion Before Changing a CI File](https://automationtester.in/blog/tools-reviews/review-yaml-conversion-before-ci), but not that `asserts` should be `assert`. For a generated eval target, the reviewing habits in [Review AI-Generated Playwright Tests Before You Merge Them](https://automationtester.in/blog/ai-in-testing/review-ai-generated-playwright-tests-before-merge) translate directly into assertions: a `not-contains` on `waitForTimeout` is one line. ## Apply this now Import the promptfoo config your team runs most often. Resolve every error, then read each `missing-var` warning against the test's description: those are the tests most likely to have been passing without testing anything. Pin a grader, save the file, and run `validate` followed by `eval`. Keep the Findings list from before the fix in the pull request, so a reviewer can see which green results were never real. ## promptfoo and Nunjucks references - [promptfoo: Assertions and metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/): assertion properties, `not-` negation, weights, test thresholds, and weight 0 passing automatically. - [promptfoo: Deterministic metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/): `contains-any` lists, `is-json` with a schema, `latency` needing `--no-cache`, `cost` thresholds. - [promptfoo: LLM Rubric](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/): the 0.0 to 1.0 score, default graders per API key, and overriding the grader. - [promptfoo: Test case configuration](https://www.promptfoo.dev/docs/configuration/test-cases/): CSV test files and the `__expected` column syntax. - [promptfoo: Command line](https://www.promptfoo.dev/docs/usage/command-line/): `validate`, `--no-cache`, `--grader` and `-r`. - [promptfoo 0.123.1 config JSON schema](https://raw.githubusercontent.com/promptfoo/promptfoo/0.123.1/site/static/config-schema.json): the assertion type list, and the closed key sets for the config, tests and assertions. - [promptfoo 0.123.1 `src/csv.ts`](https://github.com/promptfoo/promptfoo/blob/0.123.1/src/csv.ts): the `__expected` parser, its fallback to `equals`, and the 0.75 default threshold. - [promptfoo 0.123.1 `src/assertions/index.ts`](https://github.com/promptfoo/promptfoo/blob/0.123.1/src/assertions/index.ts): the “Unknown assertion type” error at run time. - [Nunjucks API](https://mozilla.github.io/nunjucks/api.html): `throwOnUndefined` defaults to `false`. - [AutomationTester.in LLM Eval Test-Case Builder](https://automationtester.in/tools/llm-eval-case-builder): the import, findings and export shown above. ## Continue reading on AutomationTester.in - [Preview a GitHub Actions Matrix Before include Adds the Wrong Jobs](https://automationtester.in/blog/tools-reviews/preview-github-actions-matrix-include-exclude) - [Review YAML Conversion Before Changing a CI File](https://automationtester.in/blog/tools-reviews/review-yaml-conversion-before-ci) - [Calibrate an LLM Judge Before You Trust Its Scores](https://automationtester.in/blog/ai-in-testing/calibrate-llm-as-judge-scores) - [Build an LLM Eval Dataset from Real Product Risks](https://automationtester.in/blog/ai-in-testing/build-llm-eval-dataset-product-risks) - [Use AI for Performance Diagnosis Without Inventing Causes](https://automationtester.in/blog/ai-in-testing/ai-performance-diagnosis-evidence-boundaries) Source: [Catch promptfoo Config Mistakes That validate Lets Through](https://automationtester.in/blog/tools-reviews/catch-promptfoo-config-mistakes-before-eval) by Shashank Rawlani.