Route Description
GET /test-cases/:testCaseId/confs Get list of test configurations
GET /test-confs/:confId Get test configuration details
POST /test-cases/:testCaseId/confs Create test configuration
HEAD /test-confs/:confId Check if test configuration exists
PUT /test-confs/:confId Update test configuration
DELETE /test-confs/:confId Delete test configuration
GET /test-confs/:confId/summary-history Get summary history
PUT /test-schedules/:scheduleId Pause/resume schedule
GET /test-confs/:confId/report.csv Export test run history to csv
POST /test-confs/:confId/share Share latest results
GET /scheduled Get list of scheduled test configurations

Overview

API routes related to test configurations. A test configuration is a collection of parameters that define how to execute a load test including the scenario, concurrency, duration, regions, and breaking points.

Get list of test configurations

Retrieve a list of all test configurations within a given test case.

GET /test-cases/:testCaseId/confs

Request Parameters

None

Sample Response

[
   {
      "id":183,
      "createdAt":"2016-03-10T00:53:12.278Z",
      "updatedAt":"2016-03-11T06:58:35.080Z",
      "name":"Demo",
      "providers":[  
         {  
            "id":33,
            "createdAt":"2016-05-23T22:14:23.088Z",
            "updatedAt":"2016-05-23T22:14:23.088Z",
            "numRegions":1,
            "params":{  
               "regions":"1"
            },
            "provider":{  
               "id":0,
               "createdAt":"2016-05-06T13:56:53.328Z",
               "updatedAt":"2016-05-06T13:56:53.328Z",
               "name":"Public Shared Grid",
               "public":true,
               "providerType":"Shared",
               "active":true
            }
         }
      ],
      "durationSecs":60,
      "iterationSleepSecs":10,
      "rampupSecs":60,
      "concurrentClients":5,
      "scriptParams":"{}",
      "isTryItOut":false,
      "active":true,
      "script":{
         "id":295,
         "createdAt":"2016-03-10T00:52:50.567Z",
         "updatedAt":"2016-03-10T00:52:59.779Z",
         "name":"Load In Chrome",
         "sourceType":"Recording",
         "sourceId":187,
         "length":3702,
         "hasInit":false,
         "hasTeardown":false,
         "scriptType":"Code"
      }
   }
]

Get test configuration details

Retrieve details about a specific test configuration.

GET /test-confs/:confId

Request Parameters

  • down: Whether to include all child objects including executions and tokens. Defaults to false.
  • up: Whether or not to include the parent, the organization details. Defaults to true.
  • offset: Offset within the executions to start the response. Defaults to 0. When combined with limit it is useful for paging through the results.
  • limit: Number of executions to return. Returns all by default or if set to -1.

Sample Response

{
   "id":183,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"Demo 12",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":60,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":5,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Create test configuration

Creates a new test configuration.

POST /test-cases/:testCaseId/confs

Request Body

  • name Required</span>: Name of the configuration.
  • scriptId Required</span>: Scenario to execute.
  • providers Required</span>: An array of provider configurations (i.e. test runners). Use the /providers route to find all available providers (aka test runners). For more details see the test runners guide.
  • profileType: The load profile type for this test. Flat is the default if not specified.
    • Flat: Rampup (rampupSecs) to a constant number of concurrent clients per region (concurrentClients) for the duration of the test. concurrentClients parameter is required. Both duration and iteration based tests are supported for this load profile type.
    • Step: Gradually increase the number of concurrent users in a step function. Specify the number of concurrent clients to start (startConcurrentClients), finish (concurrentClients), and the step size (step). The steps will be evenly distributed across the duration of your test. 50 steps is the most allowed per region. Only duration based tests are supported for this load profile type.
    • BreakingPoint: Let Testable find the maximum number of users that can concurrently execute your scenario without performance degrading below your acceptable performance standards. This means you must specify one or more breaking points to express what you consider unacceptable performance (triggers). Testable will gradually increase the number of concurrent clients until either one or more breaking points hit or your infrastructure successfully handles the maximum concurrent clients per test allowed under your account. Only duration based tests are supported for this load profile type.
  • durationSecs: Duration of the test in seconds. Either this field or iterations is required.
  • iterations: Number of times to execute the scenario on each concurrent client. Either this field or iterations is required. Only supported if the profileType is Flat.
  • iterationSleepSecs: Sleep time in seconds between iterations. Defaults to 10 seconds, minimum is 1 second.
  • rampupSecs: Time to ramp up to the desired number of concurrent clients. Defaults to 60 seconds, minimum is 0 seconds. Only supported if the profileType is Flat.
  • concurrentClients: Number of threads that will execute the scenario concurrently during the test per region. For Step load profiles this is the number of concurrent clients to finish with. Not required if the profileType is BreakingPoint.
  • startConcurrentClients: Number of concurrent clients to start the test with. Only required if the profileType is Step.
  • step: How many concurrent clients to add each time the test steps up during execution. Only required if the profileType is Step.
  • scriptParams: If any script parameters were configured, a string containing a JSON object of key/value pairs should be provided here.
  • imageDiffMode: For scenario types that support screenshot capture define the image comparison mode (None, PreviousExecution, ScenarioBaseline).
  • triggers: Also referred to as breaking points in the UI. Each trigger has a set of conditions that cause it to fire and a set of actions that should be applied in case the trigger fires. Currently any one condition being met will cause the trigger to fire and the only supported action is to stop the test execution. Required if the profileType is BreakingPoint.
  • schedules: See below for details on how to schedule your test to run later or on a recurring schedule.

Request Example:

{
   "name": "New Test Case",
   "scriptId": 295,
   "providers":[  
      {  
         "providerId":0,
         "params":{  
            "regions":"1"
         },
         "numRegions":1
      }
   ],
   "durationSecs": 120,
   "iterationSleepSecs": 10,
   "rampupSecs": 60,
   "concurrentClients": 50,
   "triggers": [
      {
         "conditions": [ {
            "metric": "firstReceivedMs",
            "metricKey": "p50",
            "isPercentage": false,
            "operator": "GTE", 
            "threshold": 1000,
            "timePeriod": "LastMinute",
            "valueType": "Absolute"
         } ],
         "actions": [ { "actionType": "StopExecution" } ]
      }
   ]
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   },
   "triggers":[  
      {  
         "id":23,
         "createdAt":"2016-09-07T05:10:14.011Z",
         "updatedAt":"2016-09-07T05:10:14.011Z",
         "conditions":[  
            {  
               "id":24,
               "createdAt":"2016-09-07T05:10:14.013Z",
               "updatedAt":"2016-09-07T05:10:14.013Z",
               "metric":"firstReceivedMs",
               "metricKey":"p50",
               "isPercentage":false,
               "timePeriod":"LastMinute",
               "operator":"GTE",
               "valueType":"Absolute",
               "threshold":1000.00
            }
         ],
         "actions":[  
            {  
               "id":22,
               "createdAt":"2016-09-07T05:10:14.017Z",
               "updatedAt":"2016-09-07T05:10:14.017Z",
               "actionType":"StopExecution"
            }
         ]
      }
   ]
}

Scheduling

Tests can have one or more schedules that either run later or on a recurring schedule using the schedules array property of the request body.

General Structure

{
   "name": "New Test Case",
   "scriptId": 295,
   ...
   "schedules": [ ... ],
}

Each schedule can be defined as follows depending on the goal:

Run Once Later

{ 
   "scheduleType": "Single", 
   "nextScheduled": 1492796529000 
}

Recurring Schedule

Minute example (Every 90 minutes):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Minute", 
   "minute": 90 
}

Hourly example (Every 2 hours at 30 minutes passed the hour):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Hour", 
   "hour": 2, 
   "minute": 30 
}

Week day example (Every week day at 13:30 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "WeekDay", 
   "hour": 13, 
   "minute": 30 
}

Daily example (Every day at 13:30 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Day", 
   "hour": 13, 
   "minute": 30 
}

Weekly example #1 (Bi-weekly on Thursday at 01:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Week", 
   "week": 2, 
   "dayOfWeek": 4, 
   "hour": 1, 
   "minute": 0 
}

Weekly example #2 (Every Monday at 01:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Week", 
   "dayOfWeek": 1, 
   "hour": 1, 
   "minute": 0 
}

Monthly example (Every 2 months on the 1st at 13:00 UTC):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Month", 
   "month": 2, 
   "hour": 13, 
   "minute": 0 
}

Schedule Exclusions

For recurring schedules you can define one or more exclusion windows in which the test should not run. If the next run falls inside any exclusion window that test run will be skipped.

Exclusions can be on the following recurring intervals: Month, Week, WeekDay, Day, Hour. All examples below assume the following recurring schedule (every 10 minutes):

{ 
   "scheduleType": "Recurring", 
   "recurringInterval": "Minute", 
   "minute": 90,
   "exclusions": [ ... ]
}

Monthly exclusion (from the 1st @ 22:00 UTC until the 3rd @ 02:00 UTC):

{
   "recurringInterval": "Month",
   "startDay": 1,
   "endDay": 3,
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Weekly exclusion (from Friday @ 17:00 UTC until Monday @ 05:00 UTC):

{
   "recurringInterval": "Week",
   "startDayOfWeek": 5,
   "endDayOfWeek": 1,
   "startHour": 17,
   "endHour": 5,
   "startMinute": 0,
   "endMinute": 0
}

Daily exclusion (from 22:00 UTC until 02:00 UTC):

{
   "recurringInterval": "Day",
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Week day exclusion (from 22:00 UTC until 02:00 UTC):

{
   "recurringInterval": "WeekDay",
   "startHour": 22,
   "endHour": 2,
   "startMinute": 0,
   "endMinute": 0
}

Hourly exclusion (from 0-15 minutes passed the hour):

{
   "recurringInterval": "Hour",
   "startMinute": 0,
   "endMinute": 15
}

Check if test configuration exists

HEAD /test-confs/:confId

Response

HTTP Status: 200 OK or 404 NOT FOUND

Update test configuration

Update the test configuration parameters. All parameters available when creating can be specified during update. To delete the associated schedule use "deleteSchedule": true in your request.

PUT /test-confs/:confId

Request Body

{
	"name": "New Name"
}

Response

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Name",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Delete test configuration

DELETE /test-confs/:confId

Response

The deleted test configuration

{
   "id":184,
   "createdAt":"2016-03-09T17:53:12.278Z",
   "updatedAt":"2016-03-10T23:58:35.080Z",
   "name":"New Test Case",
   "providers":[  
      {  
         "id":33,
         "createdAt":"2016-05-23T22:14:23.088Z",
         "updatedAt":"2016-05-23T22:14:23.088Z",
         "numRegions":1,
         "params":{  
            "regions":"1"
         },
         "provider":{  
            "id":0,
            "createdAt":"2016-05-06T13:56:53.328Z",
            "updatedAt":"2016-05-06T13:56:53.328Z",
            "name":"Public Shared Grid",
            "public":true,
            "providerType":"Shared",
            "active":true
         }
      }
   ],
   "durationSecs":120,
   "iterationSleepSecs":10,
   "rampupSecs":60,
   "concurrentClients":50,
   "scriptParams":"{}",
   "isTryItOut":false,
   "active":true,
   "script":{
      "id":295,
      "createdAt":"2016-03-09T17:52:50.567Z",
      "updatedAt":"2016-03-09T17:52:59.779Z",
      "name":"Load In Chrome",
      "sourceType":"Recording",
      "sourceId":187,
      "length":3702,
      "hasInit":false,
      "hasTeardown":false,
      "scriptType":"Code"
   }
}

Get test configuration details

Retrieves the metric summary for the most recent executions of this test configuration.

GET /test-confs/:confId/summary-history

Request Parameters

  • down: Whether to include metrics as part of each summary. Defaults to false.
  • region: Retrieve the summaries for one specific region if specified. This is the region id found in the GET /regions API. Defaults to empty (i.e. returns summary aggregated across all test regions).
  • maxExecutionId: The last test execution to include in the results. No executions after that point will be returned. Defaults to empty which means the most recent executions are returned.
  • period: What time period to query summary histories from. Possible values are recent (last 15 or limit parameter value), 7d (last 7 days), 1m (last 1 month), 3m (last 3 months). Defaults to recent if start/end not specified.
  • start: The start timestamp (Unix epoch in milliseconds, UTC). If specified, the period parameter is ignored.
  • end: The end timestamp (Unix epoch in milliseconds, UTC). Defaults to the current timestamp if start is specified.
  • limit: The number of results to return. Defaults to 15 unless a period other than recent is specified in which case it defaults to -1 (i.e. return entire history).

Sample Response

[
   {
      "id":383367,
      "createdAt":"2016-04-21T03:32:08.840Z",
      "updatedAt":"2016-04-21T03:32:08.840Z",
      "executionId":1135,
      "intervalStart":0,
      "iterationsExecuted":42,
      "count":294,
      "concurrentClients":6,
      "actualConcurrentClients":0,
      "maxConcurrentClients":6,
      "memory":100118528,
      "cpu":12.0,
      "timespan":80000,
      "metrics":[
         {
            "metricDef":"httpResponseCode",
            "metricType":"Histogram",
            "metricValueMap":{
               "200":210,
               "204":42,
               "301":42
            }
         },
         {
            "metricDef":"dnsLookupMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":2,
               "min":1,
               "p95":5,
               "max":21,
               "p99":9,
               "sd":1,
               "p50":2,
               "sum":699,
               "var":3
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"success",
            "metricType":"Counter",
            "metricValue":294,
            "metricUnits":"requests"
         },
         {
            "metricDef":"connectionOpenMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":3,
               "min":2,
               "p95":8,
               "max":23,
               "p99":11,
               "sd":2,
               "p50":3,
               "sum":1153,
               "var":4
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"bandwidth",
            "metricType":"Counter",
            "metricValue":10242207,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"httpMethod",
            "metricType":"Histogram",
            "metricValueMap":{
               "GET":294
            }
         },
         {
            "metricDef":"bytesReceived",
            "metricType":"Counter",
            "metricValue":10146405,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"bytesSent",
            "metricType":"Counter",
            "metricValue":95802,
            "metricUnits":"bytes"
         },
         {
            "metricDef":"firstReceivedMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":20,
               "min":3,
               "p95":61,
               "max":170,
               "p99":73,
               "sd":21,
               "p50":15,
               "sum":6146,
               "var":453
            },
            "metricUnits":"ms"
         },
         {
            "metricDef":"dataPacketsReceived",
            "metricType":"Counter",
            "metricValue":1645,
            "metricUnits":"packets"
         },
         {
            "metricDef":"connectionCloseMs",
            "metricType":"Timing",
            "metricValueMap":{
               "count":294,
               "mean":22,
               "min":4,
               "p95":63,
               "max":173,
               "p99":76,
               "sd":21,
               "p50":17,
               "sum":6717,
               "var":463
            },
            "metricUnits":"ms"
         }
      ]
   }
]

Pause/resume Schedule

If your test configuration has a recurring schedule, it can be paused or resumed using this endpoint. To find the scheduleId load the test configuration JSON and it can be found at schedules[0].id.

PUT /test-schedules/:scheduleId

Request Body

{
   "active": true
}

Pass active value of true to resume the schedule or false to pause it.

Result

Returns 200 status code if successful with no body.

Export test run history to csv

Export the test run history including P/F indicator, max concurrent users, p50/p99 response times, request success rate, and assertion pass rate. By default returns the last 15 runs of this test. CSV is sorted in reverse chronological order.

GET /test-confs/:confId/report.csv

Request Parameters

  • period: What time period to export test runs from. Possible values are recent (last 15 or limit parameter value), 7d (last 7 days), 1m (last 1 month), 3m (last 3 months). Defaults to recent if start/end not specified.
  • start: The start timestamp (Unix epoch in milliseconds, UTC). If specified, the period parameter is ignored.
  • end: The end timestamp (Unix epoch in milliseconds, UTC). Defaults to the current timestamp if start is specified.
  • limit: The number of results to return. Defaults to 15 unless a period other than recent is specified in which case it defaults to -1 (i.e. return entire history).
  • offset: Offset within the test run history to start the response. Defaults to 0.

Share latest results

Share the latest run of the test configuration via a public share link. Anyone who has the link can see the latest results without logging in.

POST /test-confs/:confId/share

Request Body

Optionally specify the view id and/or timezone that you want people to see the results with when accessing the share link

{
   "dashboardId": 0,
   "tz": "America/Chicago" 
}

Response

{
   "id":1773,
   "createdAt":"2022-12-12T22:52:08.400Z",
   "updatedAt":"2022-12-12T22:52:08.400Z",
   "name":"rprimspglm",
   "targetType":"GetLatestExecution",
   "targetId":1234,
   "description":"{\"dashboardId\":0,\"tz\":\"America/Chicago\"}",
   "active":true,
   "organization":{
      "id":2
   },
   "firstVersion":true
}

The name property can then be used as the public token via https://a.testable.io/p/:name to access the latest run of this test configuration if any exists. https://a.testable.io/p/:name/results.pdf to download the same as a PDF.

Get list scheduled test configurations

Retrieve list of all scheduled test configurations either for a single organization or across all organizations you have access to.

GET /orgs/:orgId/scheduled
GET /scheduled

Request Parameters

  • orgId (optional): ID of the organization

Sample Response

[
  {
    "id": 1,
    "createdAt": "2021-11-12T05:55:18.137Z",
    "updatedAt": "2021-11-12T05:55:18.137Z",
    "nextScheduled": "2021-11-15T06:55:00.000Z",
    "scheduleType": "Single",
    "active": true,
    "configuration": {
      "id": 1394,
      "createdAt": "2021-11-12T05:55:17.934Z",
      "updatedAt": "2021-11-12T05:55:18.076Z",
      "testType": "Load",
      "name": "TES-645",
      "durationSecs": 60,
      "iterationSleepSecs": 10,
      "rampupSecs": 60,
      "concurrentClients": 1,
      "profileType": "Flat",
      "scriptParams": "{}",
      "isTryItOut": false,
      "active": true,
      "hasChanged": false,
      "expectedNumRegions": 1,
      "expectedActualConcurrentClientRatio": 1,
      "outputFileRetention": "IntelligentSampling",
      "outputFileMaxCount": 1000,
      "minCredits": 0.06250,
      "onlyCaptureFailure": true,
      "imageDiffMode": "PreviousExecution",
      "manualStart": false,
      "testCase": {
        "id": 422,
        "createdAt": "2021-09-08T17:04:48.165Z",
        "updatedAt": "2021-11-10T07:11:13.940Z",
        "name": "Tar Test",
        "active": true,
        "lastRunAt": "2021-11-10T07:10:55.030Z",
        "lastRunExecutionId": 10873,
        "tags": [],
        "firstVersion": false
      },
      "executions": [],
      "script": {
        "id": 9713,
        "createdAt": "2021-09-08T17:05:03.903Z",
        "updatedAt": "2021-09-08T17:05:03.912Z",
        "name": "HTTP GET",
        "dataLength": 32,
        "hasInit": false,
        "hasTeardown": false,
        "paramsConfig": [],
        "scriptType": "Code",
        "readonly": false,
        "dataStores": [],
        "configurations": [],
        "tags": [],
        "firstVersion": true
      },
      "providers": [
        {
          "id": 4866,
          "createdAt": "2021-11-12T05:55:18.112Z",
          "updatedAt": "2021-11-12T05:55:18.112Z",
          "numRegions": 1,
          "params": {
            "regions": "1",
            "concurrentClients": "1",
            "weight": "100"
          },
          "provider": {
            "id": 1,
            "createdAt": "2017-02-27T09:03:33.102Z",
            "updatedAt": "2017-02-27T09:03:33.102Z",
            "name": "Public Shared Grid",
            "public": true,
            "providerType": "Shared",
            "metadata": {},
            "active": true,
            "locations": [],
            "firstVersion": true
          },
          "firstVersion": true
        }
      ]
    }
  }
]