Write a Script

Introduction

Define a scenario using a script for maximum flexibility and customization. Scripts are written in Javascript and execute in a sandboxed Node.js environment.

A few notes about script execution:

  1. A full execution of the script is an iteration
  2. Any network operations (e.g. http, https, websocket, net, tls) will be measured (timing, bandwidth, success, etc) and the results captured and aggregated.
  3. The script page provides a dropdown to insert an example usage of common operations.

Create a Script

There are two ways to create a script:

  1. Click the ‘New Test Case’ button on the dashboard (or Test Case -> New… in the left nav) and select ‘Script’ during the ‘Scenario’ step.
  2. If a test case already exists, click on the test case name in the dashboard or left navigation, select the ‘Scenarios’ tab, and press ‘New Scenario’. You can either start from scratch or use an existing recording as your starting point for the script.

Simple Example

In this example we call http://www.google.com in our script. This looks very simple:

http.get('http://wwww.google.com');

Example With Test Cases (Mocha.js Syntax)

Testable supports defining test cases with steps using a Mocha.js style syntax. Most features of Mocha.js are supported.

const rp = require('request-promise');
const assert = require('assert');
describe('My test suite', function() {
  it('Validate the stock symbol', async function() {
    const quote = await rp({ uri: 'http://sample.testable.io/stocks/IBM', json: true });
    assert(quote.symbol === 'IBM', 'Symbol in quote was not IBM');
  });
});

Testable Utils

Testable provides several APIs as part of the testable-utils npm module. When run locally it will print to the console. When run via Testable it integrates seamlessly with the platform. More details can be found in the README or in the various sections below.

Logging

Logging from the script shows up in the ‘Console’ section of the test results. 5 logging levels are supported: fatal, error, info, debug, and trace. Fatal logging will cause the entire test to stop. Trace logging is only captured during smoke tests.

const log = require('testable-utils').log;

log.fatal("Use this to log the error and stop the entire test execution immediately.");
log.error("Use this to log errors to display in the results");
log.info("Use this to log info statements to display in the results");
log.debug("Use this for debugging. Remember that during a load test the scenario can get executed many times!");
log.trace("Only logged during a smoke test!");

Note: Keep in mind that your script can potentially execute many times depending on the test configuration. Each Testable account has a limit on logging per test and overall storage used. Check Settings -> Test Limits to see your limits. If any limit is breached the test will immediately stop executing.

Init/Teardown

The init and teardown scripts run exactly once globally per test execution. The init script runs before the test starts and the teardown script script runs after it completes.

The syntax and modules available are exactly the same as during the test.

Environment Variables

When Testable runs your Node.js script the following environment variables are passed into your test:

  • OUTPUT_DIR: A local directory to output any files you want to collect as part of the test results.
  • TESTABLE_EXECUTION_ID: The unique ID of this test case execution.
  • TESTABLE_REGION_NAME: The name of the region in which the test plan is executing (e.g. aws-us-east-1).
  • TESTABLE_GLOBAL_CLIENT_INDEX: A unique identifier for each virtual user within the execution. Starts at 0.
  • TESTABLE_REGIONAL_CLIENT_INDEX: A unique identifier for each virtual user within this region of the execution. Starts at 0.
  • TESTABLE_CONCURRENT_CLIENTS: Total number of virtual users globally configured for this test execution.
  • TESTABLE_ITERATION: Per virtual user, which iteration of the test scenario you are currently on. Starts at 0.
  • TESTABLE_ITERATION_ID: A globally unique uuid that is unique per virtual user scenario iteration.
  • TESTABLE__UID: A unique id (across the test execution) for a virtual user scenario iteration.
  • TESTABLE_CHUNK_ID: Each test runner within a test execution is assigned a unique chunk ID.
  • TESTABLE_UNIQUE_INDEX: A unique index per virtual user test iteration. Starts at 0.

Script Parameters

In some situations you may want to use the same script across multiple Test Configurations. In this case there may be certain parameters that need to be different for different configurations.

Parameter values can be accessed in your code using environment variables. For example the value for parameter Abc is accessible as environment variable PARAM_ABC.

Read more about scenario parameters here.

Loading Additional Modules

If your script requires an NPM module that is not listed at the top of this guide, go ahead and try to use it and see if it is already available. For example:

const mysql = require('mysql');

Additional modules are not downloaded into our Node.js environment by default. When your script requires it, the module is installed and loaded dynamically for your use.

The full list of whitelisted modules is always changing. When writing your script select “Available NPM Modules” from the dropdown in the upper right to see the currently available full list. If you don’t see your module there, please email support@testable.io to have it added.

All modules must be available on the public NPM module registry at https://npmjs.org to be eligible currently. Support for private repositories will be considered in the future.

Local Testing

Our testable-utils library is available on NPM and supports local execution. This allows you to run the tests locally before uploading them to Testable.

const utils = require('testable-utils');
const dataTable = utils.dataTable;
const log = utils.log;

// etc etc etc

Handing Async Flows

If you are using a Node.js module in your script that has async flows you need to indicate to Testable the start and finish of that flow. The following modules are exceptions where Testable instruments the module to handle the async flow: async, http, https, request, net, ws, socketio, engineio, tls, setTimeout, setInterval. In those cases you do not need to worry about it. For other cases you have two options:

Option 1: Via a Promise

If the last statement in your script results in a Promise, Testable will wait for that Promise to finish before considering your script done.

new Promise(function (resolve, reject) {
  console.log('Lets resolve now');
  resolve();
})
Option 2: Explicitly with the execute() utility

Use the execute() utility that Testable provides to indicate when your code is finished executing.

const execute = require('testable-utils').execute;
execute(function(finished) {
  someModule.funcWithAsyncFlow('123', function() {
    // async callback
    console.log('do some stuff');
    finished();
  });
});

Reading from a CSV

To provide different parameters for each iteration of the test script use this module. See the upload data page for more details about the API.

const testable = require('testable-utils').dataTable;
const row = await dataTable.open('demo.csv').next();
http.get('http://sample.testable.io/stocks/' + rows[0].data['SYMBOL']);

Capture Custom Metrics

To capture custom metrics in your test script use this module. See the custom metrics page for more details about the API.

const results = require('testable-utils').results;

results().counter('My Custom Counter', 2, 'items');
results().histogram('Response Codes', '123');
results().timing('Custom Timing', 225, 'ms');
results().metered('Browser Memory', os.hostname(), 2543253, 'bytes');

Stopwatch

Convenience function that executes your code, times how many milliseconds it takes, and captures it as a custom timing metric.

API

const stopwatch = require('testable-utils').stopwatch;
stopwatch(code, metricName[, resource]);

Example

stopwatch(function(done) {
  // some operations go here
  done();
}, 'My Custom Timer');

Live Manual Event

You can manually trigger an event while a test is running from the test results page (action menu => Send Live Event) or our API. Your script can listen for this event and perform an action in response. This is useful if you want to have all the virtual users perform an action at the exact same time for example. The event name/contents can be whatever you want.

Example

Listen for an event, my-event where the contents are a symbol. In response request a stock quote and “finish” the test script. When run locally or in a smoke test, fire the event immediately.

const request = require('request');
const testableUtils = require('testable-utils');
const events = testableUtils.events;
const execute = testableUtils.execute;
const fireNow = testableUtils.isLocal || testableUtils.isSmokeTest;

execute(function(finished) {
  events.on('my-event', function(symbol) {
    request.get('http://sample.testable.io/stocks/' + symbol);
    finished();
  });
});

if (fireNow)
  events.emit('my-event', 'MSFT');

NPM and Node.js Modules

Each section below details a Node.js or NPM module or package that is available for use in a script.

The following Node.js/NPM modules are always available to use during script execution:

  1. Request
  2. HTTP
  3. HTTPS
  4. Net
  5. TLS
  6. WebSocket
  7. socket.io-client
  8. engine.io-client
  9. Lodash
  10. Math
  11. moment
  12. setTimeout/clearTimeout
  13. process
  14. util
  15. url
  16. uuid
  17. jsonfile
  18. har-replay

Testable allows for additional whitelisted NPM modules to be downloaded on demand. See the Loading Additional Modules section for more details.

Request Module

All options provided by the request NPM module are supported.

An example GET request:

request.get('http://sample.testable.io/stocks/IBM');

An example POST request:

const req = request.post('http://httpbin.org/post', { 
  headers: {
    'X-Test-Header': 'blablabla'
  }
}, function(err, res, body) {
  log.info('BODY: ' + body);
});
req.write('request body test');
req.end();

In the above example we call POST http://httpbin.org/post with a X-Test-Header header and request body test as the body. See the module documentation for the full range of options.

HTTP Module

All options provided by the client side of the Node.js HTTP module are supported. This includes http.get() and http.request().

An example POST request:

const req = http.request({ 
  hostname: 'httpbin.org', 
  path: '/post', 
  method: 'POST', 
  headers: {
    'X-Test-Header': 'blablabla'
  }
}, function(res) {
  res.on('data', function (chunk) {
    log.info('BODY: ' + chunk);
  });
});
req.write('request body test');
req.end();

In the above example we call POST http://httpbin.org/post with a X-Test-Header header and request body test as the body. See the Node.js documentation for full details of the options.

To make an HTTP request without Testable tracking and reporting metrics (e.g. reporting the start of a test to your servers):

httpNoTracking.get('https://myserver.com');
HTTPS Module

All options provided by the client side of the Node.js HTTPS module are supported. This includes http.get() and http.request().

https.get('https://www.google.com');

To make an HTTPS request without Testable tracking and reporting metrics (e.g. reporting the start of a test to your servers):

httpsNoTracking.get('https://myserver.com');
Net Module

All options provided by the client side of the Node.js Net module are supported.

const client = net.connect({ host: 'sample.testable.io', port: 8091 }, function() {
  // connected!
  client.write('test echo message');
});

client.on('data', function(data) {
  log.info(data.toString());
  client.end();
});

client.on('end', function() {
  log.info('disconnected from server');
});
TLS Module

All options provided by the client side of the Node.js TLS module are supported.

const wss = new WebSocket("wss://wss.websocketstest.com/service");

wss.on('open', function open() {
  wss.send('echo,test message');
});

wss.on('message', function(data, flags) {
  log.info(data);
  wss.close();
});

wss.on('error', function(error) {
  log.error(error);
  wss.close();
});
WebSocket Module

All options provided by the client side of the ‘ws’ NPM package module are supported.

In the below example we test the sample HTTP/WS service.

const ws = new WebSocket("ws://sample.testable.io/streaming/websocket");

ws.on('open', function open() {
  ws.send('{ "subscribe": "IBM" }');
});

ws.on('message', function(data, flags) {
  log.info(data);
  ws.close();
});

ws.on('error', function(error) {
  log.error(error);
  ws.close();
});
Socket.io Client Module

See the socket.io-client documentation for the full set of options.

The below example connects to our sample Socket.io echo service.

const socket = socketio('http://sample.testable.io:5811');
socket.on('connect', function(){
  log.info('connected');
  socket.emit('message', 'This is a test');
});
socket.on('event', function(data){
  log.info(data);
  socket.close();
});
socket.on('disconnect', function(){
  log.info('disconnected');
});
Engine.io Client Module

See the engine.io-client documentation for the full set of options.

The below example connects to our sample Engine.io echo service.

const socket = engineio('http://sample.testable.io:5812');
socket.on('open', function(){
  log.info('opened');
  socket.send('This is a test'); 
  socket.on('message', function(data){
    log.info(data);
    socket.close();
  });
  socket.on('close', function(){
    log.info('closed');
  });
});
Lodash Module

See the Lodash documentation for all the functions it supports.

const symbols = ['IBM', 'MSFT', 'AAPL'];
_.forEach(symbols, function(symbol) {
  http.get('http://sample.testable.io/stocks/' + symbol);
});
Math Module

Any function of the Math object can be used in a script.

const rand = Math.random();
if (rand > 0.5) {
  // do one thing here
} else {
  // do something else
}
moment Module

Any function of the momentjs can be used in a script.

log.info("Current timestamp: " + moment().valueOf());
setTimeout/clearTimeout

Use the setTimeout() function to add a delay to your script. The below example delays the entire script by 100ms.

setTimeout(function() {
  // your scenario code here

}, 100); // 100ms delay
setInterval/clearInterval

Use the setInterval() function to run a function regularly. The below runs some code every 100ms.

const handle = setInterval(function() {
  // your code here

}, 100); // every 100ms
 
// some point later
clearInterval(handle);
Process Module

The following functions/properties in the Node.js process module are supported:

  1. process.uptime()
  2. process.hrtime()
  3. process.memoryUsage()
  4. process.env
  5. process.arch
  6. process.platform
log.info(util.inspect(process.memoryUsage()));
Util Module

All functions in the Node.js util module are supported.

const test = { a: b, c: d};
log.info(util.inspect(test));
URL Module

All functions in the Node.js url module are supported.

UUID Module

All functions in the NPM uuid module are supported.

jsonfile Module

All read functions in the NPM jsonfile module are supported.

const jsonfile = require('jsonfile');
const myJsonObj = jsonfile.readFileSync('myUploaded.json');
har-replay Module

An NPM module, har-replay, which supports reading and replaying the contents of a HTTP Archive (HAR). See the README for more details.

const harReplay = require('har-replay');
harReplay.load('myUploaded.har');

Execution Info

During execution the info object provides information on the current execution context. This object is unique per concurrent user and is accessible as a global variable in your script. It includes:

  1. Execution: Details of the execution including id, concurrent clients, duration/iterations, etc
  2. Agent: Unique identifier for the agent on which this test iteration is executing
  3. Region: Region (id, name, description) where the test iteration is executing
  4. Chunk: Within each region, the execution is broken into chunks. The details of the current chunk are provided here.
  5. Client: A unique number for each concurrent client within this chunk. Uses zero based index.
  6. Global Client Index: A unique number for each concurrent client globally across the test. Uses zero based index.
  7. Regional Client Index: A unique number for each concurrent client within this region of the test (e.g. AWS N. Virginia). Uses zero based index.
  8. Iteration: Within each concurrent client each iteration is assigned an increasing id.
  9. Unique ID: To get a unique natural ID corresponding to this iteration, use info.currentId. This will return a string that combines all of the above values into a unique natural key.
  10. Output Directory: Directory to output files that you want to capture as part of the test results. This directory will be captured on a couple of test iterations per minute, not necessarily on every one.
  11. Context: A way to maintain state across iterations of a single concurrent user. Any property that is part of this object will get passed between one concurrent user’s iterations of the script. Any properties assigned to this object must be serializable to JSON (i.e. no functions). See the maintaining state across iterations section for more details. 10 Expected Finish Timestamp: The timestamp (millisecond unix epoch) when the test is expected to finish if it is configured to run for a duration. When a test is configured for a certain number of iterations or during a smoke test this field will be -1.

It is also accessible via require('testable-utils').info for better local testing compatibility.

An example of the info structure:

{ expectedFinishTimestamp: 1553095840154,
  iteration: 98,
  client: 2,
  globalClientIndex: 12,
  regionalClientIndex: 6,
  chunk: 
   { id: 47,
     executionType: 'Main',
     agent: '238b9add-e342-4e3f-af53-131ea9a866c7',
     createdAt: '2015-09-28T17:45:20.187Z',
     updatedAt: '2015-09-28T17:45:20.188Z',
     startedAt: '2015-09-28T17:45:20.512Z',
     iterations: 5,
     concurrentClients: 5,
     chunkIndex: 2,
     globalConcurrentClientIndex: 10,
     regionalConcurrentClientIndex: 5
   },
  agent: '238b9add-e342-4e3f-af53-131ea9a866c7',
  execution: 
   { id: 48,
     createdAt: '2015-09-28T17:45:10.611Z',
     updatedAt: '2015-09-28T17:45:17.120Z',
     startedAt: '2015-09-28T17:45:13.519Z',
     iterations: 5,
     concurrentClients: 1
   },
  region: 
   { id: 1,
     createdAt: '2015-08-11T22:03:34.761Z',
     updatedAt: '2015-08-11T22:03:34.761Z',
     name: 'aws-us-east-1',
     public: true,
     latitude: 39.0436,
     longitude: -77.4878,
     description: 'AWS N. Virginia',
     active: true 
   },
   outputDir: '/tmp/some/path/here',
   context: 
   { authToken: 'example-abcdef'
   }
 }

Maintaining State Across Iterations

As mentioned above, the execution info object provides a mechanism for passing state between each concurrent user’s iterations of a script.

It is accessible in your script via info.context and by default is an empty object. Use this object to maintain session state like authentication details.

if (!info.context.myAuthToken) {
  // get the auth token on the first iteration of this concurrent user
  info.context.myAuthToken = 'abcdef';
}

// use the auth token
console.log('Token: ' + info.context.myAuthToken);