Self Hosted AWS Test Runner Setup

Test Runner instances generate load for your test. These instances can self-hosted within your AWS account. Testable will start the EC2 instances in your account as part of running your test and terminate it upon test completion.

In order to make this work you need to grant Testable access to your AWS account. The steps to do that are detailed in this guide.

Step 1: Create an AWS account

If you do not already have an AWS account, go to aws.amazon.com to create one.

Step 2: Define an IAM policy

Before we can setup a user and access key we need to define an IAM policy with the permissions required for Testable.

Login to the AWS Console for the IAM service and click on the Policies section and then Create Policy.

Use the following JSON for the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Testable",
            "Action": [
                "ec2:*",
                "iam:ListRoles",
                "iam:PassRole",
                "iam:ListInstanceProfiles"
                "iam:GetRole"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ]
}

Then simply give it a name (e.g. Testable) and press Create Policy.

Create IAM Policy

Step 3: Create an IAM user

Next we create a new IAM user within the AWS Console for the IAM service. Click Users on the left and then Add User.

Give it a name (e.g. testableio) and select Programmatic Access.

Add IAM User

On the Permissions step choose Attach existing policies directly and then select the policy we created in step 2.

Add IAM User Permissions

Note the access key id and secret access key once the user is created. Both of these values are required on the Testable side of this setup.

Step 4: Create a Test Runner Source

Login to your Testable account and go to Account => Test Runner Sources and press the Add On Demand Source button in the upper left.

Use the access key id and secret access key from step 3 to setup the test runner.

Create Test Runner

And that's it! You will now see this test runner in the list when you create a new test configuration.

Add

FAQ

How do the test runners communicate and coordinate with Testable? My test is getting stuck provisioning infrastructure.

Test runners must be able to connect OUT to agents.testable.io on port 443. Make sure that your VPC and subnet are configured with an internet gateway to make this possible.

If you run a test and it gets stuck on the "provisioning infrastructure" step this is almost certainly the issue. To test connectivity consider spinning up a small EC2 instance in the same VPC/subnet that you configured on Testable, SSH to it, and try "telnet agents.testable.io 443" to verify connectivity to Testable:

ubuntu@ip-172-2-1-235:~$ telnet agents.testable.io 443
Trying 34.231.181.104...
Connected to alpha-coordinator-elb-860939053.us-east-1.elb.amazonaws.com.
Escape character is '^]'.