Test Runners

• Introduction
• AWS Cloud
  • Per Test: Testable VPC
  • Per Test: Your VPC
  • Long Running: Public Shared Grid
  • Long Running: Your VPC
    • Testable AWS AMI
    • Unlimited Testing AWS Marketplace AMI
• Azure Cloud
  • Per Test: Testable Virtual Network
  • Per Test: Your Virtual Network
  • Long Running: Your Virtual Network
• GCP
  • Per Test: Testable Virtual Network
  • Per Test: Your Virtual Network
  • Long Running: Your Virtual Network
• Self-Hosted: Your Infrastructure
• Windows Test Runners
  • Windows test runner limitations
• Test Configuration
• Management
  • Register AWS/Azure/GCP Account as Source
  • Update Cloud Account Settings
  • Delete Test Runner Source

Introduction

Test runners run your test and simulate virtual users as defined in your scenario. Test can be allocated across multiple test runners in one or more regions around the world. The test runner agent can run on any machine where Docker is installed. Currently test runners can be run in the following ways:

• AWS Cloud: Run your test from any AWS region. There are several ways to do this:
  1. Per Test: Testable VPC: Configure your test to spin up test runner EC2 instances in Testable’s VPC automatically. The instances will be terminated at the end of the test.
  2. Per Test: Your VPC: Configure your test to spin up test runner EC2 instances in your VPC automatically. The instances will be terminated at the end of the test.
  3. Long Running: Public Shared Grid: A Testable managed grid of test runners available in some AWS regions and shared by all customers. Useful for running small tests quickly without needing to wait for EC2 instances to provision.
  4. Long Running: Your VPC: Use our Testable Agent {ts} or Testable Windows Agent {ts} AMIs or the Marketplace AMI to start test runner instances in your VPC and register them as test runners.
• Azure Cloud: Run your test from any Azure location. There are several ways to do this:
  1. Per Test: Testable Virtual Network: Configure your test to spin up test runner VMs in Testable’s virtual network automatically. The instances will be terminated at the end of the test.
  2. Per Test: Your Virtual Network: Configure your test to spin up test runner VMs in your virtual network automatically. The instances will be terminated at the end of the test.
  3. Long Running: Your Virtual Network: Use our VM image in the Azure Marketplace to start test runner instances in your virtual network and register them as test runners.
• GCP: Run your test from any GCP location. There are several ways to do this:
  1. Per Test: Testable Virtual Network: Configure your test to spin up test runner VMs in Testable’s virtual network automatically. The instances will be terminated at the end of the test.
  2. Per Test: Your Virtual Network: Configure your test to spin up test runner VMs in your virtual network automatically. The instances will be terminated at the end of the test.
• Self-Hosted: Your Infrastructure: Spin up test runners anywhere you have Docker installed and can connect OUT to agents.testable.io on port 443.

AWS Cloud

You can run your test from any AWS region. Either spin up and register test runners instances yourself before a test run or setup your test configuration so that Testable does it automatically for you.

Per Test: Testable VPC

Test runner EC2 instances will be spun up at the start of a test and automatically terminated at the end. Test runners are private to that one specific test run.

After selecting an AWS region, choose Source = AWS - Testable Account to have the EC2 test runner instances hosted within Testable’s VPC. The Test Runners section will appear in the configuration where you can customize the instance type and number of instances. Testable makes a default automatic recommendation as well.

Per Test: Your VPC

Test runner EC2 instances will be spun up at the start of the test and automatically terminated at the end. Test runners are private to that one specific test run.

After selecting an AWS region, choose Source => Register New AWS Account the first time to grant Testable the access required to spin up EC2 instances in your VPC.

The Test Runners section will appear in the configuration where you can customize the instance type and number of instances. Testable makes a default automatic recommendation as well.

Once you select your AWS account in the Source list additional fields will appear to choose the VPC, subnet, key pair, and elastic IPs.

Long Running: Public Shared Grid

A Testable managed grid of test runners distributed across 7 AWS regions and shared by all customers. Useful for running small tests quickly without needing to wait for EC2 instances to provision.

Long Running: Your VPC

Test runner EC2 instances can also be spun up before a test runs, registered with Testable, and then used to execute one or more tests. There are two options on how to do this.

Testable AWS AMI

Use the Testable Agent {ts} AMI available on AWS.

Start EC2 instances using the latest version of the AMI called Testable Agent {ts} via the AWS Console or API. Alternatively, on Testable go to Organization => Test Runner Sources => Self-Hosted and press the Register Test Runner button. A dropdown will appear near the top where you can select the desired AWS region and open the EC2 launch wizard with the right AMI automatically selected for you.

Register Manually

1. Login to your Testable account and go to Organization => Test Runner Sources => Self-Hosted.
2. Press the Register Test Runner button.
3. Enter a comma separated list of instance ids or public host names. Test runners will take a few minutes to appear here after launch.
4. Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Locations => Self-Hosted => [region]). Tests will be evenly distributed across the test runners in the region.

Register Automatically

In order for the instance to register automatically with your organization and region you should provide it the following user data during the instance creation process where the region name can be anything and TESTABLE_KEY is an API key for your organization found under Org Management => API Keys:

Linux

#!/bin/bash

sudo tee -a /etc/environment > /dev/null << EOF
TESTABLE_REGION=my-region-name
TESTABLE_KEY=xxx
EOF

Windows

<powershell>
[System.Environment]::SetEnvironmentVariable("TESTABLE_KEY", "xxx", [System.EnvironmentVariableTarget]::Machine)
[System.Environment]::SetEnvironmentVariable("TESTABLE_REGION", "my-region-name", [System.EnvironmentVariableTarget]::Machine)

$env:TESTABLE_KEY="xxx"
$env:TESTABLE_REGION="my-region-name"
</powershell>

Once the instance is up and running you will see it under Org Management => Test Runner Sources => Self-Hosted => Your Region Name.

View all registered test runners in a region by clicking its name.

AWS Marketplace AMI

Similar to the above option (Testable AWS AMI) except you pay per hour via the AWS marketplace and can run as many tests as you want as long as the tests run in a region with all test runners being unlimited.

Azure Cloud

You can run your test from any Azure location. Testable will automatically start the VMs at the beginning of your test and delete them at the end.

Run your test in a dedicated per test Azure VM scale set, using either Testable’s account or self-hosting within your own.

Per Test: Testable Virtual Network

After selecting an Azure location, select Source = Azure - Testable Account to have the VM scale set test runners hosted within Testable’s virtual network. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances, etc. Testable makes a default automatic recommendation as well.

Per Test: Your Virtual Network

Configure your own Azure account to self-host the VM scale set test runners. Testable will automatically terminate the scale set once the test is finished in that region. Once you select an Azure location, register your Azure account via Source => Register New Azure Account. Be sure to read our Self-Hosted Azure Test Runner Setup guide for detailed requirements.

Once setup you can select your Azure account as the source and then choose the resource group, network, subnet, and storage account.

Long Running: Your Virtual Network

Test runner VMs can also be spun up before a test runs, registered with Testable, and then used to execute one or more tests.

Go to Azure Marketplace and search for Testable and select either our Linux or Windows image to launch your VM.

Register Manually

1. You’ll need the VM resource ID. This can be found in the Azure portal by going to the VM details -> Properties -> Resource ID.
2. Login to your Testable account and go to Organization => Test Runner Sources => Self-Hosted.
3. Press the Register Test Runner button.
4. Enter a comma separated list of resource ids from step 1. Test runner VMs will take a few minutes to appear here after launch.
5. Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Locations => Self-Hosted => [region]). Tests will be evenly distributed across the test runners in the region.

Register Automatically

In order for the instance to register automatically with your organization and region you should provide it the following custom data during the instance creation process where the region name can be anything and TESTABLE_KEY is an API key for your organization found under Org Management => API Keys:

Linux

#!/bin/bash

sudo tee -a /etc/environment > /dev/null << EOF
TESTABLE_REGION=my-region-name
TESTABLE_KEY=xxx
EOF

Windows

<powershell>
[System.Environment]::SetEnvironmentVariable("TESTABLE_KEY", "xxx", [System.EnvironmentVariableTarget]::Machine)
[System.Environment]::SetEnvironmentVariable("TESTABLE_REGION", "my-region-name", [System.EnvironmentVariableTarget]::Machine)

$env:TESTABLE_KEY="xxx"
$env:TESTABLE_REGION="my-region-name"
</powershell>

Once the instance is up and running you will see it under Org Management => Test Runner Sources => Self-Hosted => Your Region Name.

View all registered test runners in a region by clicking its name.

GCP

You can run your test from any GCP location. Testable will automatically start the VMs at the beginning of your test and delete them at the end.

Run your test in a dedicated per test GCP VM instance, using either Testable’s account or self-hosting within your own.

Per Test: Testable Virtual Network

After selecting a GCP location, select Source = GCP - Testable Account to have the VM instance test runners hosted within Testable’s virtual network. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances, etc. Testable makes a default automatic recommendation as well.

Per Test: Your Virtual Network

Configure your own GCP account to self-host the VM test runners. Testable will automatically terminate the VMs once the test is finished in that region. Once you select a GCP location, register your GCP account via Source => Register New GCP Account. Be sure to read our Self-Hosted GCP Test Runner Setup guide for detailed requirements.

Once setup you can select your GCP account as the source and then choose the zone, network, subnetwork.

Long Running: Your Virtual Network

Test runner VMs can also be spun up before a test runs, registered with Testable, and then used to execute one or more tests.

Copy Image To Your Project

Before you can launch your VMs you need to copy the image(s) to your project:

1. Go to https://api.testable.io/gcp/image-info to get the latest image names.
2. Run the following via the GCP CLI: gcloud compute --project=[your-project] images create testable-agent-xxx --source-image=[image-name-from-step-1] --source-image-project=[project-name-from-step-1]

Once you have the image you can launch a VM from that image. In order to register it with Testable you have two options:

Option 1: Register Manually

1. Login to your Testable account and go to Organization => Test Runner Sources => Self-Hosted.
2. Press the Register Test Runner button.
3. Enter a comma separated list of VM ids or public host names. Test runners will take a few minutes to appear here after launch.
4. Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Locations => Self-Hosted => [region]). Tests will be evenly distributed across the test runners in the region.

Option 2: Register Automatically

In order for the instance to register automatically with your organization and region you should provide it the following startup script during the instance creation process where the region name can be anything and TESTABLE_KEY is an API key for your organization found under Org Management => API Keys:

Linux

#!/bin/bash

sudo tee -a /etc/environment > /dev/null << EOF
TESTABLE_REGION=my-region-name
TESTABLE_KEY=xxx
EOF

Windows

<powershell>
[System.Environment]::SetEnvironmentVariable("TESTABLE_KEY", "xxx", [System.EnvironmentVariableTarget]::Machine)
[System.Environment]::SetEnvironmentVariable("TESTABLE_REGION", "my-region-name", [System.EnvironmentVariableTarget]::Machine)

$env:TESTABLE_KEY="xxx"
$env:TESTABLE_REGION="my-region-name"
</powershell>

Once the instance is up and running you will see it under Org Management => Test Runner Sources => Self-Hosted => Your Region Name.

View all registered test runners in a region by clicking its name.

Self-Hosted: Your Infrastructure

A self-hosted region can also be setup that is shared across tests witin your organization only. This can be a useful way to leverage your own data centers and/or existing cloud infrastructure for running load tests. It also allows you to use the same test runners across multiple tests and avoid the wait for per test infrastructure on each test.

Run on your own infrastructure using Docker. Test runners are grouped by region name so that your team can have several different pools for different types of tests (e.g. load tests vs continuous integration). Whatever region names you provide when using the docker run command will appear in the list. The test runners need to be able to connect OUT to agents.testable.io but Testable does not need to connect IN to your data center.

Windows Test Runners

The standard test runners runs in a lightweight Ubuntu 22.04 LTS based Docker container (phusion/baseimage). Our AMI (for AWS) or VM image (for Azure) is simply an Ubuntu virtual machine with Docker installed that runs the Testable Docker container on startup.

In some cases, when running Selenium tests, you may want to run the test on the Windows OS. We provide a version of our test runner that runs natively on Windows.

It is packaged as an AMI for AWS (Name = Testable Windows Agent [timestamp]). To run a test on Windows you must choose Runtime Requirements = Windows in your Selenium scenario setup. This will cause Testable to use the Windows test runner image when launching a Test Runner.

Windows test runner limitations

No Public Shared Grid Windows test runners are available currently. This means you cannot run your Windows tests on our public shared grid which includes smoke tests.

If you want to smoke test your scenarios, this is a good way to do it. Launch an instance using the Windows AMI and follow the steps for associating it with a private region above. Then either make that private region your default smoke testing region (Org Management => Settings => Smoke Test Region) OR choose Smoke Test in Region from the dropdown menu in the upper right hand corner of a Selenium scenario.

Test Configuration

When creating a new test configuration you must select one or more locations to run your test.

So for example you could run your scenario on our shared public grid, inside your data center, and using per test AWS EC2 instances all at the same time.

Management

Go to Org Management -> Test Runner Source to manage your sources.

Register AWS/Azure Account as Source

In the AWS or Azure tabs, press the Register Account as Source button to register your account.

Update Cloud Account Settings

Update your private cloud account details for AWS or Azure by clicking the configure icon next to the test runner source name.

Delete Test Runner Source

Delete your AWS or Azure account as a source by clicking the trash icon below the test runner source name.