Concepts

• Test Case
• Scenario
• Test Configuration
• Test Runners
• Triggers
• Analysis
• Alerts
• API
• Enterprise/Portability

Testable is focused on simplifying testing without sacrificing flexibility and power. The key concepts are described here.

Test Case

A test case holds a collection of scenarios and configurations that are related. This can be thought of as a way to group tests by project within a single organization.

Scenario

The scenario defines what each virtual user in the test should do during the test.

For example a scenario for an HTTP based REST API might involve:

1. Send GET / to http://mywebsite.com/api
2. Wait 100ms for a reply, check the status code to ensure a 200 OK.

There are a numerous ways to define a scenario:

• Create a Recording: Testable will dynamically generate a URL that you can use to record all traffic to and from your service as you use the gateway URL (i.e. a recording man in the middle proxy).
• Write a Node.js script: Write a script using the Javascript language running within a sandboxed Node.js environment to execute more complex scenarios. You can start from scratch or convert a Recording into a script as a starting point.
• Run a JMeter Test Plan: JMeter, a popular source tool, can be used to define the scenario. Then simply use Testable to execute that test plan at scale.
• Run a Gatling Simulation: Gatling, a Scala DSL based open source tool, can be used to define the scenario. Then simply use Testable to execute that simulation at scale.
• Run a Locust test: Locust, a Python based open source tool, can be used to define the scenario. Then simply use Testable to execute at scale.
• Write a Webdriver.io script: Write a Webdriver.io Selenium script for Nodejs. Each virtual user runs wdio against a test runner local selenium instance.
• Write a Protractor script: Write a Protractor script. Each virtual user runs protractor against a test runner local selenium instance.
• Write Selenium Javascript code: Write Selenium Javascript code. Each virtual user runs this code in Node.js against a test runner local selenium instance.
• Write Selenium Java code: Write Selenium Java code. Each virtual user runs the Java code on a JVM against a test runner local selenium instance.
• Run a Serenity BDD project: Upload a Serenity BDD project. Each virtual user will run the Serenity tests on a JVM against a test runner local selenium instance.
• Run an OpenFin application: Test an OpenFin application using any of the supported Selenium bindings. Each virtual user will launch the OpenFin application with Chromedriver running locally on the test runner.
• Replay a HAR file: Each virtual user replays a HAR file.
• Load Url: The simplest option. Ask Testable to either HTTP GET your URL or simulate loading your URL in a Chrome browser.
• Write a PhantomJS or SlimerJS script: Write a script for PhantomJS, a headless WebKit browser implementation or for SlimerJS a headless Gecko (Firefox) browser implementation.

Test Configuration

A test configuration defines how to simulate your scenario. How many virtual users, which locations to generate traffic from, the test duration, success criteria, scheduling, network conditions, filters, etc. The configuration can be run many times and results of each test run can be analyzed individually or compared. Trend analysis is performed automatically by Testable. Multiple test configurations can run the same scenario with different load profiles.

Test Runners

Test Runners do exactly that: actually run your test on whatever machines you configure in your test. There are several options:

• AWS Cloud: Run your test from any AWS region. There are several ways to do this:
  1. Public Shared Grid: A Testable managed grid of test runners available in some AWS regions and shared by all customers. Useful for running small tests quickly without needing to wait for EC2 instances to provision.
  2. Per Test: Testable VPC: Configure your test to spin up test runner EC2 instances in Testable’s VPC automatically. The instances will be terminated at the end of the test.
  3. Per Test: Your VPC: Configure your test to spin up test runner EC2 instances in your VPC automatically. The instances will be terminated at the end of the test.
  4. Long Running: Your VPC: Use our Testable Agent {ts} AMI to start test runner instances in your VPC and register them as test runners.
• Azure Cloud: Run your test from any Azure location. There are several ways to do this:
  1. Per Test: Testable Virtual Network: Configure your test to spin up test runner VMs in Testable’s virtual network automatically. The instances will be terminated at the end of the test.
  2. Per Test: Your VPC: Configure your test to spin up test runner EC2 instances in your VPC automatically. The instances will be terminated at the end of the test.
• GCP Cloud: Run your test from any GCP location. There are several ways to do this:
  1. Per Test: Testable Virtual Network: Configure your test to spin up test runner VMs in Testable’s virtual network automatically. The instances will be terminated at the end of the test.
  2. Per Test: Your VPC: Configure your test to spin up test runner VM instances in your VPC automatically. The instances will be terminated at the end of the test.
• Self-Hosted: Your Infrastructure: Spin up test runners anywhere you have Docker installed and can connect OUT to agents.testable.io on port 443.

See the test runners guide for more details.

Triggers

Tests can be triggered in a variety of ways:

1. Manual: Manually start the test execution from the Testable website.
2. Trigger: A URL that accepts an HTTP POST request which kicks off the test. Find the trigger URL on the test configuration page -> Triggers tab. This provides a simple way to trigger test execution that can be used in Continuous Integration and other dev ops tools.
3. Continuous Integration : Execute a test case when code is checked in by using a Trigger URL as part of your continuous integration pipeline or by using our Simple API.
4. Remote Selenium Grid: Our platform can be used as a remote selenium grid to run your functional tests. Our remote webdriver url is at https://selenium.testable.io/wd/hub. Each session will correspond to a functional test launched on a test runner. Read our remote Selenium guide for more details.
5. Remote DevTools Browser: Point either Playwright or Puppeteer at our remote devtools endpoint. This will launch a browser on one of our test runners and create a tunnel to it. Available at wss://agents.testable.io/devtools. Read our remote devtools guide for more details.

Analysis

The results of a test case execution are only useful if you can analyze and report on them. Testable will offer a few different ways to do this:

1. Testable Website: Analysis, reporting, and sharing functionality on test results can be found via our web tool.
2. API: Access results of test cases via our API and analyze them however you want. Summary stats (i.e. mean, 95% latency, etc) are also calculated and provided for you via this route.
3. Third Party Integration: Push the results to a third party product for analysis and reporting. Currently only New Relic is supported. Stay tuned for more details on other integration partners.

Alerts

Test results are only really useful if you can define the conditions under which you want to be alerted of potential issues. Set success criteria and enable notifications on test failure. Use our platform to detect scale issues but also to monitor your infrastructure more generally.

API

Testable was designed from the ground up using the same API that is provided to customers. This means every feature you find on the website is available via the API. We hope this allows customers to build and integrate the solutions they need around the Testable core product.

Enterprise/Portability

Fully On Prem: We offer a fully on-prem solution where the entire Testable stack is installed on premises with no connections back to Testable’s cloud.

Hybrid: We also offer a hybrid solution where the test runners are on premises and Testable hosts everything else.