Skip to main content

Pattern Matching Evaluators

Pattern Matching Evaluation is a method used to assess the performance of LLMs by identifying specific patterns within the output generated by the model. In Agenta, to perform pattern matching evaluation you can make use of the following evaluators:

  • Regex Test
  • Starts with
  • Ends with
  • Contains
  • Contains Any
  • Contains All

Regular Expression​

Regular Expressions (Regex) are sequences of characters that define search patterns, often used for pattern matching within strings. The Regex Test evaluator checks if the generated answer matches a regular expression pattern.

The evaluator takes the regex pattern, and whether to match or not match for that pattern.

Here are some examples:

OutputRegexMatch/MismatchEvaluator Output
The iPhone 6 has a 1024px screen.*iphone.*matchTrue
The Samsung galaxy has a 1024px screen.*Samsung.*mismatchFalse

Starts With​

Starts With evaluator checks if the output starts with a specified prefix, considering case sensitivity based on the settings.

Ends With​

Ends With evaluator checks if the output ends with a specified suffix, considering case sensitivity based on the settings.

Contains​

Contains evaluator checks if the output contains a specified substring, considering case sensitivity based on the settings.

Contains Any​

Contains Any evaluator checks if the output contains any of the specified substrings from a comma-separated list, considering case sensitivity based on the settings.

Contains All​

Contains All evaluator checks if the output contains all of the specified substrings from a comma-separated list, considering case sensitivity based on the settings.