Test Runners

Introduction

Test runners run your test and simulate virtual users as defined in your scenario. Test can be allocated across multiple test runners in one or more regions around the world. The test runner agent can run on any machine.

There are two sources of test runners:

  1. Shared: A shared grid of test runners can run one or more tests concurrently and can consistent of many machines. Testable provides a public shared grid for all Testable customers. You can also setup a self-hosted shared grids for tests within your organization only.

  2. On Demand: Testable can spin up a set of test runners specifically for one test only. Once initialized they will simulate all the virtual users defined in your test. Once finished Testable will terminate all test runners.

Add

Shared Test Runners

Shared grids execute your test immediately assuming there is capacity instead of waiting for infrastructure to be allocated.

Public Shared Grid

Testable provides a public shared grid distributed across the globe on the infrustructure of various cloud providers. Tests can be run on this grid but keep in mind it is shared by all Testable users.

Public Grid

Self-Hosted Shared Grid

A self-hosted shared grid can also be setup that is shared across tests witin your organization only. This can be a useful way to leverage your own data centers and existing infrastructure for running load tests. It also allows you to use the same test runners across multiple tests and avoid the wait for on demand infrastructure on each test.

Private Grid

There are three ways to start test runners in a private shared grid:

Option 1: Run on your own infrastructure with Docker

Run on your own infrastructure using Docker. Test runners are grouped by region name so that your team can have several different pools for different types of tests (e.g. load tests vs continuous integration). Whatever region names you provide when using the docker run command will appear in the list.

Option 2: Testable AWS AMI

Use the Testable Agent AMI available on AWS.

  1. Start EC2 instances using the latest version of the AMI called Testable Agent via the AWS Console or API. Alternatively, on Testable go to Organization => Test Runner Sources and press the Register Shared Instances button. A dropdown will appear near the top where you can select the desired AWS region and open the EC2 launch wizard with the right AMI automatically selected for you.
  2. Login to your Testable account and go to Organization => Test Runner Sources.
  3. Press the Register Shared Instances button.
  4. Enter a comma separated list of instance ids or public host names from step 1. Test runners will take a few minutes to appear here after launch.
  5. Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Test Runners => My Shared Grid => [region]). Tests will be evenly distributed across the test runners in the region.

Register Shared Test Runner

View all registered test runners in a region by clicking its name.

Test Runners in Region

Option 3: Unlimited Testing AWS Marketplace AMI

Similar to option 2 above (Testable AWS AMI) except you pay per hour via the AWS marketplace and can run as many tests as you want as long as the tests run in a region with all test runners being unlimited.

On Demand Test Runners

On demand test runners are started by Testable at the start of a test run and are dedicated to your test only. Once the test finishes, these test runners are terminated.

AWS

Add

Run your test on dedicated on-demand AWS EC2 instances, using either Testable's account or self-hosting within your own.

AWS Option 1: EC2 Instances Within the Testable AWS Account

Select AWS - Testable Account to have the EC2 test runner instances hosted within Testable's account. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances. Testable makes a default automatic recommendation as well.

AWS On Demand Public

AWS Option 2: Self Hosted EC2 Instances Within Your AWS Account

Configure your own AWS account to self-host the EC2 test runner instances. Testable will automatically terminate the instances once the test is finished in that region. Configure your AWS account in the Locations widget under Source => Register New AWS Account. Be sure to read our Self-Hosted AWS Test Runner Setup guide for detailed requirements.

Once setup you can select your AWS account as the source and then choose the VPC/subnet.

Self-Hosted AWS On Demand

Azure

Add

Run your test in a dedicated on-demand Azure VM scale set, using either Testable's account or self-hosting within your own.

Azure Option 1: VM Scale Set Within the Testable Azure Account

Select Azure - Testable Account as your source to have the VM scale set test runners hosted within Testable's account. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances, etc. Testable makes a default automatic recommendation as well.

Azure On Demand Public

Azure Option 2: Self Hosted VM Scale Set Within Your Azure Account

Configure your own Azure account to self-host the VM scale set test runners. Testable will automatically terminate the scale set once the test is finished in that region. Configure your Azure account in the Locations widget under Source => Register New Azure Account. Be sure to read our Self-Hosted Azure Test Runner Setup guide for detailed requirements.

Once setup you can select your Azure account as the source and then choose the resource group, network, subnet, and storage account.

Self-Hosted Azure On Demand

Test Configuration

When creating a new test configuration you must select one or more locations to run your test.

So for example you could run your scenario on our shared public grid, inside your data center, and using on demand AWS EC2 instances all at the same time.

Management

Go to Organization -> Test Runner Source to manage your sources.

Test Runner List

Add On Demand Source

Press the Add On Demand Source button in the upper left to add a new test runner. Currently we support both AWS and Azure.

Add New Test Runner

Update On Demand Source Settings

Update your private cloud account details for AWS or Azure by clicking the configure icon next to the test runner source name.

Delete Test Runner Source

Delete your AWS or Azure account as a source by clicking the trash icon next to the test runner source name.