Test Runners

Introduction

Test runners run your test and simulate virtual users as defined in your scenario. Test can be allocated across multiple test runners in one or more regions around the world. The test runner agent can run on any machine.

There are two sources of test runners:

  1. Shared: A shared grid of test runners can run one or more tests concurrently and can consistent of many machines. Shared grids can be private to your organization or public for sharing across all Testable customers.

  2. On Demand: Testable can spin up a set of test runners specifically for one test only. Once initialized they will simulate all the virtual users defined in your test. Once finished Testable will terminate all test runners.

Add

Shared Test Runners

Shared grids execute your test immediately assuming there is capacity instead of waiting for infrastructure to be allocated.

Public Shared Grid

Testable provides a public shared grid distributed across the globe on the infrustructure of various cloud providers. Tests can be run on this grid but keep in mind it is shared by all Testable users.

Public Grid

Private Shared Grid

A private shared grid can also be setup that is shared across tests witin your organization only. This can be a useful way to leverage your own data centers and existing infrastructure for running load tests. It also allows you to use the same test runners across multiple tests and avoid the wait for on demand infrastructure on each test.

Private Grid

The My Shared Grid test runner source is automatically created the first time you register or run a private shared test runner.

There are three ways to start test runners in a private shared grid:

Option 1: Run on your own infrastructure

Run on your own infrastructure using Docker. Test runners are grouped by region name so that your team can have several different pools for different types of tests (e.g. load tests vs continuous integration). Whatever region names you provide when using the docker run command will appear in the list.

Option 2: Testable AWS AMI

Use the Testable Agent AMI available on AWS.

  1. Start EC2 instances using the latest version of the AMI called Testable Agent via the AWS Console or API. Alternatively, on Testable go to Organization => Test Runner Sources and press the Register Shared Instances button. A dropdown will appear near the top where you can select the desired AWS region and open the EC2 launch wizard with the right AMI automatically selected for you.
  2. Login to your Testable account and go to Organization => Test Runner Sources.
  3. Press the Register Shared Instances button.
  4. Enter a comma separated list of instance ids or public host names from step 1. Test runners will take a few minutes to appear here after launch.
  5. Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Test Runners => My Shared Grid => [region]). Tests will be evenly distributed across the test runners in the region.

Register Shared Test Runner

View all registered test runners in a region by clicking its name.

Test Runners in Region

Option 3: Unlimited Testing AWS Marketplace AMI

Similar to option 2 above (Testable AWS AMI) except you pay per hour via the AWS marketplace and can run as many tests as you want as long as the tests run in a region with all test runners being unlimited.

On Demand Test Runners

On demand test runners are started by Testable at the start of a test run and are dedicated to your test only. Once the test finishes, these test runners are terminated.

AWS

Add

Run your test on dedicated on-demand AWS EC2 instances, using either Testable's account or self-hosting within your own. Other cloud platforms will be added in the future.

Option 1: EC2 Instances Within the Testable AWS Account

Select AWS - Testable Account to have the EC2 test runner instances hosted within Testable's account.

AWS On Demand Public

Option 2: Self Hosted EC2 Instances Within Your AWS Account

Configure your own AWS account to self-host the EC2 test runner instances. Testable will automatically terminate the instances once the test is finished in that region. Click the New... option if you are setting up your AWS account for the first time and make sure to read our Self-Hosted AWS Test Runner Setup guide for detailed requirements.

Self-Hosted AWS On Demand

Notice the extra settings:

  • VPC: The AWS VPC where you want the test runners to run. Make sure the VPC chosen can connect the https://agents.testable.io.
  • Subnet: The subnet within the chosen VPC where you want the test runners to run.
  • Tags: Additional tags to add to any EC2 instance started. These can be specified either in the test configuration or a default set when creating/updating the test runner configuration.
  • Key Pair:Choose a key pair you have access to so that you can connect and monitor the instances we spun up as part of your test.

Test Configuration

When creating a new test configuration you must select one or more test runner sources to run your test. The same test runner source can be configured multiple times within a single configuration as long as you stay within your account limits.

So for example you could run your scenario on our shared public grid, inside your data center, and using on demand AWS EC2 instances all at the same time.

Management

Go to Organization -> Test Runner Source to manage your sources.

Test Runner List

Add On Demand Source

Press the Add On Demand Source button in the upper left to add a new test runner. Currently the only on demand source is AWS.

Read the Self-Hosted AWS Test Runner Setup guide for step-by-step instructions.

Add New Test Runner

Update On Demand Source Settings

Update your private AWS credentials, tags, and default VPC settings by clicking the configure icon next to the test runner source name.

Delete Test Runner Source

Delete your AWS account as a source by clicking the trash icon next to the test runner source name.