To create a new automated test run, go to the Runs module, click “New Run” at the top right of your screen and select “Automated”. The Testlio platform automatically names your run by default “run XXX”, but you can alter the name to anything that makes sense for you.

Setting up the run consists of up to 5 steps depending on the type of application that you are testing:

Our automated testing supports 3 options:

Choose the option that matches your application.

When testing mobile applications, you can either upload the build file or choose a recently uploaded build from the Testlio build repository.

We support both iOS and Android builds (.ipa or .apk file types, respectively, no other file types are supprted)

Note: if you need to test both Android and iOS apps, you need to create two separate runs. One for iOS (.ipa) and other for Android (.apk). You can´t do both tests on the same run.

When testing web applications or websites, you don’t need to upload a build. The URL of your website needs to be defined in your test scripts instead.

Configure your testing by selecting the framework you need from the drop-down. We currently support Appium Java TestNG, Appium Java JUnit, Appium Python, Appium Node frameworks.

You can attach your pre-compiled script test package either by uploading a .zip file from your device or by selecting a previously uploaded package.

Note: We currently only support .zip file types for script packages and .yml files for script configurations.

Our device farm provider Amazon allows custom script configuration files to be used, but we have provided an easy option to use the default setting which should accommodate most cases.

Tick the video box if you want a continuous video screen capture of your testing session.

Next, you can select several devices or browsers from a list. Please note, by default, the number of devices is limited to 2 per family (2 iOS + 2 Android + 2 Web browsers). Also, the devices will be displayed regarding the build you have uploaded (e.g. if you uploaded an apk build, will only see Android devices) or regarding the application you have chosen in the first run configuration step.

You can choose several configurations that the device would be booted up with, such as:

In this step, you can define the time out for the runs. This protects your budget from any problems that may occur during the test execution in the device farm. For example, let’s imagine that some issues have caused the script to go into an infinite loop. The timeout prevents this from using up all of the processing budget by killing the execution when the time you allocate has been exceeded. This will always happen, so be mindful of setting a reasonable time so that your test execution isn’t unnecessarily cut short.

As such, the time limit multiplied by the devices gives you an estimation of the total potential processing that would be consumed. However, utilization is always affected by real processing time rather than timeouts, so if the scripts finish ahead of the timeout, the utilization is only affected by the actual amount it took.

After finishing your configuration, you can press “start run” and confirm to start your run.

Important: This means that we are locking this work, which incurs a cost for you. After the work has been locked, we cannot the testing activity from taking place anymore.

Our system has device slots that determine the number of devices that you can select per run. In addition, the slot number determines your parallelization capacity.

Let’s imagine that you have purchased 2 device slots and create 3 runs, adding 2 devices in each with a timeout of 1 hour per each device. Your parallelization capacity is also determined by your slot amount. Meaning that in this scenario, if you start all of the three runs at the same time, the 2 devices of the first run can start processing, but the other runs will be queued up. Only 2 devices across any number of runs can process in parallel because you have purchased two slots.

After all prior steps are done, you are able to see the results in a matter of minutes.

See our help article on working with results for more detail.