Self Hosted AWS Test Runner Setup

Test Runner instances generate load for your test. These instances can self-hosted within your AWS account. Testable will start the EC2 instances in your account as part of running your test and terminate it upon test completion.

In order to make this work you need to grant Testable access to your AWS account. The steps to do that are detailed in this guide.

Step 1: Create an AWS account

If you do not already have an AWS account, go to aws.amazon.com to create one.

Step 2: Define an IAM policy

Before we can setup a user and access key we need to define an IAM policy with the permissions required for Testable.

Login to the AWS Console for the IAM service and click on the Policies section and then Create Policy.

Use the following JSON for the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Testable",
            "Action": [
                "ec2:*",
                "iam:GetRole",
                "kms:Encrypt",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ]
}

A few notes:

  1. The KMS actions are only required if you want to use your own KMS to encrypt/decrypt test parameters and files.
  2. The IAM role is only required if you are using spot instances. We use it to look up the spot fleet role to use.

Then simply give it a name (e.g. Testable) and press Create Policy.

Create IAM Policy

Spot Requests

Testable uses a spot fleet request for provisioning spot instances. For this to work you need to first setup an IAM role named aws-ec2-spot-fleet-tagging-role in your AWS account. You can read more about how to set this up at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-requests.html#spot-fleet-prerequisites.

Step 3: Create either a Cross-Account Role or an IAM user

Option 1: Cross-Account Role (Preferred)

Next we create a role that grants access to Testable’s AWS account to perform the actions associated with the policy created in Step 2.

Testable’s AWS Account: 220197662125.

Add IAM Cross Acct Role

You can optionally choose to setup an External ID for extra security which you can specify when setting things up in Testable.

Next choose the policy we created in step 2 and assign it to the role.

Finally give the role a name and create it. Go to the role details page and copy the Role ARN. You will need this to setup the access in Testable.

Option 2: Create an IAM User

Next we create a new IAM user within the AWS Console for the IAM service. Click Users on the left and then Add User.

Give it a name (e.g. testableio) and select Programmatic Access.

Add IAM User

On the Permissions step choose Attach existing policies directly and then select the policy we created in step 2.

Add IAM User Permissions

Note the access key id and secret access key once the user is created. Both of these values are required on the Testable side of this setup.

Step 4: Create a Test Runner Source

Login to your Testable account and go to Org Management => Test Runner Sources and select tab AWS and click on the Register AWS Account as Source tile.

If you used a cross account role, choose Access Type = Cross Account Role and then specify the Role ARN.

Create Test Runner

If you created an IAM user then choose Access Type = Credentials and use the access key id and secret access key from step 3 to setup the test runner.

Create Test Runner

And that’s it! You will now see this in the source list when you create a new test configuration and select an AWS region.

Add

FAQ

How do the test runners communicate and coordinate with Testable? My test is getting stuck provisioning infrastructure.

Test runners must be able to connect OUT to agents.testable.io on port 443.

If you run a test and it gets stuck on the “provisioning infrastructure” step this is almost certainly the issue. To test connectivity consider spinning up a small EC2 instance in the same VPC/subnet that you configured on Testable, SSH to it, and try “telnet agents.testable.io 443” to verify connectivity to Testable:

ubuntu@ip-172-2-1-235:~$ telnet agents.testable.io 443
Trying 34.231.181.104...
Connected to alpha-coordinator-elb-860939053.us-east-1.elb.amazonaws.com.
Escape character is '^]'.

There are two ways to ensure the instances can access the internet:

  1. Ensure the VPC subnet in which you choose to launch test runners is public and connected to an internet gateway.

  2. If your instance sits in a private subnet with no route to the internet, make sure that a NAT gateway has been setup to enable the test runner instances to access the internet.