On Premises Test Execution
- Execute a Test Case In Your Region
- Flash enabled Docker image
Testable test runners are responsible for executing test cases and reporting back the results. Shared runners are provided in regions across the globe for your use. In certain cases it makes sense for users to have their own private runners. These situations include:
- Your product sits behind a firewall and is not reachable from outside that firewall.
- Your test requires IP address whitelisting. The global shared agents are dynamically allocated and removed so a fixed range of IPs cannot be guaranteed.
- Ensure complete isolation from other user's test cases.
- Leverage spare capacity on your infrastructure.
Docker must be installed before you can start the agent. Instructions on how to install docker can be found at https://docs.docker.com/engine/installation/.
Create an API Key
Login to Testable and go to Settings -> Api Key. Create a new API key. You will need this to run the agent.
The test runner will attempt to connect to https://agents.testable.io (port 443) to receive work and report on results. Make sure outbound network access is available from the host.
A quick way to check is to run:
telnet agents.testable.io 443 and ensure it connects successfully.
docker run -e AGENT_REGION_NAME=[region-name] -e AGENT_KEY=[api-key] -v /logs --privileged --security-opt seccomp:unconfined -d testable/agent:alpha
That's it. The region name can be anything you want. This name will appear in the Regions multi-select when you create or update your test configuration and select the My Shared Grid test runner. The AGENT_KEY is the API key you created earlier (Settings -> Api Keys).
docker-compose.yml if using Docker Compose:
testable: image: testable/agent:alpha network: bridge environment: - AGENT_REGION_NAME=[region-name] - AGENT_KEY=[api-key] volumes: - /logs security_opts: - seccomp:unconfined privileged: true
Full set of accepted environment variables:
AGENT_REGION_NAME(required): The name of the region.
AGENT_KEY(required): Api key to authenticate your agent with (create api keys at Settings -> Api Keys)
AGENT_LOG_LEVEL: The logger level. Choices are (error, warn, info, verbose, debug). Defaults to info.
AGENT_LOG_CONSOLE: Whether or not to log to the console. Defaults to false.
AGENT_CONCURRENT_CLIENTS: Number of concurrent clients the agent can execute at any one time. Defaults to 500.
AGENT_REGION_DESCRIPTION: The description that appears with the region name when configuring test cases. Defaults to the region name if not specified.
AGENT_UPDATE_LAT_LONG: A boolean that indicates whether or not to update the location of your agent at runtime. Defaults to false. If true, Testable will geo locate your IP address. On the test results, your agent will appear at this location on the map.
AGENT_LATITUDE: The latitude at which your agents are located. If set along with AGENT_LONGITUDE, your region's location will be updated to these values and the geo location of your IP address ignored.
AGENT_LONGITUDE: The longitude at which your agents are located. If set along with AGENT_LATITUDE, your region's location will be updated to these values and the geo location of your IP address ignored.
AGENT_TRACE_ENABLED: Whether to enable or disable tracing for all tests executed on this test runner. Traces capture all request and response headers and body for some network requests (i.e. HTTP or websocket). In some environments this can lead to sensitive data being sent to Testable which is undesirable. Set to true by default.
PUBLIC_HOSTNAME: Optionally specify the hostname of the server running the docker container. This is important if you run multiple test runners on one server so that Testable does not count available memory/cpu multiple times.
To apply resource constraints to the container (i.e. memory/cpu) see this Docker guide. Otherwise the test runner container can use as much of a given resource as the host’s kernel scheduler allows.
Logging is written to /logs/agent.log within the container. Mount the
/logs volume if you would like to access the log from outside of Docker. 10 days worth of logs are kept in the container using a daily rolling strategy.
Create a new test case, and during the Configuration step select My Shared Grid as a Test Runner. You should see your region in the list. Select it and when your test starts it will execute on one of your test runners.
Note that you can have multiple regions of your own each with multiple agents.
To upgrade to the latest version of the test runner software:
docker pull testable/agent:alpha
The default Testable Docker image,
testable/agent:alpha, does not have the Flash plugin installed.
To test a Flash enabled website, use the
testable/agent-flash:alpha image instead.
I don't see my region in the list?
Try reloading the Testable page and it should appear as a region to select for your test configuration. If it still doesn't appear, check the outbound network access to agents.testable.io.
My tests keep crashing when run on my test runner.
If you set a
-m or --memory option with
docker run it is possible you have assigned the container a limited amount of memory that is insufficient to run your test. Use
docker inspect [container-id] to check the memory settings of your container.
I use Selenium Webdriver.io + Chrome and the browser is crashing.
There are two potential issues here:
1. Shared Memory
Chrome relies heavily on shared memory and Docker by default does not always provide much shared memory to the container.
To check this issue first run
ipcs -lm on the instance. Check the output for
max total shared memory (kbytes).
Next run the same command on your Docker container:
docker exec [container-id] ipcs -lm and observe the output for
max total shared memory (kbytes). If the shared memory in your container is lower this is likely an issue.
Recent version of docker support a setting for shared memory:
--shm-size 4G. If that does not work you can try mapping shared memory as a volume:
-v /dev/shm:/dev/shm or
-v /run/shm:/dev/shm depending on the location of shared memory on the host. Run
mount -t tmpfs to find out the location of shared memory on your host looking for
tmpfs on /XXX/shm type tmpfs (rw,nosuid,nodev) in the output.
/XXX/shm would be the shared memory location on your host.
2. Docker security settings
On some Linux installations (e.g. Ubuntu 16.04) the default security settings for your docker container are not sufficient for Chrome to launch.
We can fix this by ensuring our docker run command includes
--security-opt seccomp:unconfined. Check out this for a more restricted security profile that should be sufficient for Chrome.
Has your container image passed an independent security scan?
Yes. Our test runner image is scanned by Anchore.io and the latest results can be found here.