- Shared Test Runners
- On Demand Test Runners
- Windows Test Runners
- Test Configuration
Test runners run your test and simulate virtual users as defined in your scenario. Test can be allocated across multiple test runners in one or more regions around the world. The test runner agent can run on any machine.
There are two sources of test runners:
Shared: A shared grid of test runners can run one or more tests concurrently and can consistent of many machines. Testable provides a public shared grid for all Testable customers. You can also setup a self-hosted shared grids for tests within your organization only.
On Demand: Testable can spin up a set of test runners specifically for one test only. Once initialized they will simulate all the virtual users defined in your test. Once finished Testable will terminate all test runners.
Shared grids execute your test immediately assuming there is capacity instead of waiting for infrastructure to be allocated.
Testable provides a public shared grid distributed across the globe on the infrustructure of various cloud providers. Tests can be run on this grid but keep in mind it is shared by all Testable users.
A self-hosted shared grid can also be setup that is shared across tests witin your organization only. This can be a useful way to leverage your own data centers and existing infrastructure for running load tests. It also allows you to use the same test runners across multiple tests and avoid the wait for on demand infrastructure on each test.
There are three ways to start test runners in a private shared grid:
Run on your own infrastructure using Docker. Test runners are grouped by region name so that your team can have several different pools for different types of tests (e.g. load tests vs continuous integration). Whatever region names you provide when using the
docker run command will appear in the list.
Use the Testable Agent AMI available on AWS.
- Start EC2 instances using the latest version of the AMI called Testable Agent via the AWS Console or API. Alternatively, on Testable go to Organization => Test Runner Sources and press the Register Shared Instances button. A dropdown will appear near the top where you can select the desired AWS region and open the EC2 launch wizard with the right AMI automatically selected for you.
- Login to your Testable account and go to Organization => Test Runner Sources.
- Press the Register Shared Instances button.
- Enter a comma separated list of instance ids or public host names from step 1. Test runners will take a few minutes to appear here after launch.
- Choose the test runners to register as well as the desired region name. This is the region name that you can then choose in your test configuration (Locations => Self-Hosted => [region]). Tests will be evenly distributed across the test runners in the region.
View all registered test runners in a region by clicking its name.
Similar to option 2 above (Testable AWS AMI) except you pay per hour via the AWS marketplace and can run as many tests as you want as long as the tests run in a region with all test runners being unlimited.
On demand test runners are started by Testable at the start of a test run and are dedicated to your test only. Once the test finishes, these test runners are terminated.
Run your test on dedicated on-demand AWS EC2 instances, using either Testable's account or self-hosting within your own.
Select AWS - Testable Account to have the EC2 test runner instances hosted within Testable's account. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances. Testable makes a default automatic recommendation as well.
Configure your own AWS account to self-host the EC2 test runner instances. Testable will automatically terminate the instances once the test is finished in that region. Configure your AWS account in the Locations widget under Source => Register New AWS Account. Be sure to read our Self-Hosted AWS Test Runner Setup guide for detailed requirements.
Once setup you can select your AWS account as the source and then choose the VPC/subnet.
Run your test in a dedicated on-demand Azure VM scale set, using either Testable's account or self-hosting within your own.
Select Azure - Testable Account as your source to have the VM scale set test runners hosted within Testable's account. The Test Runners section will appear in the configuration where you can customize the instance type, number of instances, etc. Testable makes a default automatic recommendation as well.
Configure your own Azure account to self-host the VM scale set test runners. Testable will automatically terminate the scale set once the test is finished in that region. Configure your Azure account in the Locations widget under Source => Register New Azure Account. Be sure to read our Self-Hosted Azure Test Runner Setup guide for detailed requirements.
Once setup you can select your Azure account as the source and then choose the resource group, network, subnet, and storage account.
The standard test runners runs in a lightweight Ubuntu 16.04 LTS based Docker container (phusion/baseimage). Our AMI (for AWS) or VM image (for Azure) is simply an Ubuntu virtual machine with Docker installed that runs the Testable Docker container on startup.
In some cases, when running Selenium tests, you may want to run the test on the Windows OS. We provide a version of our test runner that runs natively on Windows.
It is packaged as an AMI for AWS (Name = Testable Windows Agent [timestamp]). To run a test on Windows you must choose Runtime Requirements = Windows in your Selenium scenario setup. This will cause Testable to use the Windows test runner image when launching a Test Runner.
Limitation 1: No Public Shared Grid Windows test runners are available currently. This means you cannot run your Windows tests on our public shared grid which includes smoke tests.
Limitation 2: If you launch an EC2 instance using our Windows AMI (Name = Testable Windows Agent [timestamp]) you must include the following User Data:
<powershell> Restart-Computer </powershell>
If you want to smoke test your scenarios, this is a good way to do it. Launch an instance using the Windows AMI and follow the steps for associating it with a private region above. Then either make that private region your default smoke testing region (Org Management => Settings => Smoke Test Region) OR choose Smoke Test in Region from the dropdown menu in the upper right hand corner of a Selenium scenario.
Limitation 3: Windows test runners can only be used for Selenium tests at the moment.
When creating a new test configuration you must select one or more locations to run your test.
So for example you could run your scenario on our shared public grid, inside your data center, and using on demand AWS EC2 instances all at the same time.
Go to Organization -> Test Runner Source to manage your sources.
Press the Add On Demand Source button in the upper left to add a new test runner. Currently we support both AWS and Azure.
Update your private cloud account details for AWS or Azure by clicking the configure icon next to the test runner source name.
Delete your AWS or Azure account as a source by clicking the trash icon next to the test runner source name.