Success Criteria + Breaking Points

• Introduction
• Getting Started
• General Notes
• Criteria Types
• Metrics
  • Counters
  • Timings
  • Histograms

Introduction

Define a set of criteria for what you consider a successful test outcome. For example: Median Response Time < 1000ms.

All criteria are evaluted when the test finishes. Criteria that are marked as “Break on fail” are setup as breaking points. These criteria are evalauted in real-time during the test and as soon as one is no longer met the test is automatically stopped.

Success criteria and breaking points can be configured per test configuration as well as a default set for your entire organization under Org Management => Settings. Different default criteria can be setup per scenario type (i.e. Node.js script, JMeter, Gatling, Locust, Selenium, etc).

Any metric can be used in building criteria including custom user defined metrics.

Getting Started

While creating or editing a test configuration there is a Success Criteria + Breaking Point section.

Add new criteria by selecting a common template from the dropdown or Custom… to create your own.

General Notes

1. Criteria that are set to “break on fail” are only monitored after the ramp up period of your test or after at least 1 minute.
2. If your scenario is a script that creates custom metrics, the custom metrics will only be available in the dropdown while creating criteria AFTER the first execution of your test.

Criteria Types

There are 4 types of criteria:

1. Active value (breaking points only): Monitor the value of the specified metric during the most recent minute of your test execution.
2. Value during this test: Monitor the value of the specified metric across the entire test execution.
3. Change minute over minute (breaking points only): Monitor the percentage change in the specified metric from the previous minute to the most recent minute of your test execution.
4. Change from previous test runs: Monitor the percentage change in the specified metric from a weighted average of up to 10 recent test executions to this current test execution. If there are less than 10 test executions for your configuration, all executions are included. If this is the first execution, no breaking point of this type will be triggered.

Metrics

Testable captures a bunch of standard metrics related to response time, concurrency, throughput, and success rate. All metrics are available for use in breaking points. See the metrics glossary for a more precise definition of each metric. Any custom metric is also available for monitoring.

Counters: For counter metrics (e.g. Success) you can monitor the metric itself or the metric as a percent of total requests made during the test or time interval.

Timings: For timing metrics (e.g. Response Time) several aggregations are available including percentiles, count, standard deviation, max, min, mean, and median. All these aggregators can be monitored in your criteria.

Histograms: For histograms (e.g. Http Response Code) a metric is available for each bucket and the bucket as a percent of the total sum across the all histogram buckets. So for example, you can monitor the number of 200 HTTP Response Codes or the percent of all HTTP responses that had a 200 status.