Self Hosted AWS Test Runner Setup

Test Runner instances generate load for your test. These instances can self-hosted within your AWS account. Testable will start the EC2 instances in your account as part of running your test and terminate it upon test completion.

In order to make this work you need to grant Testable access to your AWS account. The steps to do that are detailed in this guide.

Step 1: Create an AWS account

If you do not already have an AWS account, go to to create one.

Step 2: Define an IAM policy

Before we can setup a user and access key we need to define an IAM policy with the permissions required for Testable.

Login to the AWS Console for the IAM service and click on the Policies section and then Create Policy.

Use the following JSON for the policy:

    "Version": "2012-10-17",
    "Statement": [
            "Sid": "Testable",
            "Action": [
            "Effect": "Allow",
            "Resource": [

Then simply give it a name (e.g. Testable) and press Create Policy.

Create IAM Policy

Step 3: Create an IAM user

Next we create a new IAM user within the AWS Console for the IAM service. Click Users on the left and then Add User.

Give it a name (e.g. testableio) and select Programmatic Access.

Add IAM User

On the Permissions step choose Attach existing policies directly and then select the policy we created in step 2.

Add IAM User Permissions

Note the access key id and secret access key once the user is created. Both of these values are required on the Testable side of this setup.

Step 4: Create a Test Runner Source

Login to your Testable account and go to Org Management => Test Runner Sources and press the Add On Demand Source button in the upper left.

Use the access key id and secret access key from step 3 to setup the test runner.

Create Test Runner

And that's it! You will now see this in the source list when you create a new test configuration and select an AWS region.



How do the test runners communicate and coordinate with Testable? My test is getting stuck provisioning infrastructure.

Test runners must be able to connect OUT to on port 443.

If you run a test and it gets stuck on the "provisioning infrastructure" step this is almost certainly the issue. To test connectivity consider spinning up a small EC2 instance in the same VPC/subnet that you configured on Testable, SSH to it, and try "telnet 443" to verify connectivity to Testable:

ubuntu@ip-172-2-1-235:~$ telnet 443
Connected to
Escape character is '^]'.

There are two ways to ensure the instances can access the internet:

  1. Any EC2 instance with a public IP address can access the internet. Testable will use the default "enable auto-assign public IPv4 address" setting for the chosen subnet when launching instances. If you enable this setting then all launched test runner instances will have a public IP address and be able to connect to the internet.

  2. If you do not want the instances to have a public IP address, make sure that your VPC and subnet are configured with an internet gateway.