This FAQ includes the following sections:

What is Testlio Automation?

The Testlio Platform provides core test automation functionality and can integrate with third-party tools to create, run, and analyze automated tests. Testlio Automation allows developers to increase application quality, time to market, and coverage by testing and interacting with real Android and iOS devices in the Cloud. Users can upload their app, test scripts, and run automated tests in parallel across many real and racked devices, getting results, screenshots, and video in minutes.

Who should use Testlio Automation and why?

Testlio’s automated testing capabilities are designed for engineering and product teams who are building, testing, and supporting mobile apps and need to align their testing strategy to accelerate the feedback loop to developers, shorten release cycles and, deliver the quality that their customers expect. Testlio’s automated testing capabilities bring together the freelance talent, strategic services, platform functionality, and technology integrations that organizations need to speed time to market, increase quality, and efficiently scale testing

What types of apps does Testlio Automation support?

Testlio Automation supports native and hybrid Android, iOS, and web apps, and cross-platform apps including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks.

Does Testlio Automation use simulators or emulators?

Testlio Automation tests are run on real devices. The devices are a mixture of OEM and carrier-branded devices.

Slots: What is and how does it work?

Device slots correspond to concurrency. By default Testlio Platform users have 2 slots available per family (i.e. Android or iOS). For instance, if client/user purchase 5 automated test Android device slots and schedule a run on 20 Android devices, Testlio Automation Platform will execute the tests on up to 5 devices at a time until all tests are completed on the selected devices. Purchasing more slots would enable users getting results faster. Clients can ask more slots to the services team, anytime (note: a new price will be calculated).

Private device: What is and how does it work?

A private device is a physical instance of a phone or tablet that is exclusive to client account. Private devices can have custom, static configurations and run custom OS images. Each device is deployed on client behalf and removed at the end of the subscription. Clients can ask for private devices to the services team, anytime (note: a new price will be calculated).

How do I get started with Testlio Automation?

Please see our automated run setup article.

Which browsers are supported for testing web applications?

Your web applications will be tested in Chrome on Android and Safari on iOS.

What is the maximum file size for apps and tests?

Testlio Automation supports files up to 4 GB.

Do I need to instrument my app or supply source code?

No instrumentation or source code is required to use the built-in tests. Android apps can be submitted as-is. iOS apps should be built with “iOS Device” as the target instead of a simulator.

Do you modify my app?

On iOS, we replace the embedded provisioning profile with a wildcard profile and resign the app. If you provide it, we will add auxiliary data to the application package before installation so the data will be present in your app’s sandbox. Resigning the iOS app results in the removal of certain entitlements. This includes App Group, Associated Domains, Game Center, HealthKit, HomeKit, Wireless Accessory Configuration, In-App Purchase, Inter-App Audio, Apple Pay, Push Notifications, and VPN Configuration & Control.

On Android, we resign the app. This may break functionality that depends on the app signature, such as the Google Maps Android API. It may also trigger anti-piracy and anti-tamper detection available from products such as DexGuard. For built-in tests, we may modify the manifest to include permissions required to capture and save screenshots.

What types of builds are allowed?

For iOS we allow only .ipa files. For Android we allow .APK files but you can also install multiple .APKs. You only need to use the .adb command in the test spec file (important note: this type of builds are not the standard, meaning that there are several limitations during the tests and results. Some limitations are the Android chats are not displayed and granular videos and screenshots are not performed very well).

Which devices are available in Testlio Automation? How do you select the devices in your fleet?

Testlio Automation has a large selection of Android, iOS, and Fire OS devices. We add popular new devices as they are released by manufacturers. We also add new devices as new OS versions are released. See here the list of available devices.

How do I select devices? Can I retest on the same device?

For Automated Testing, devices are selected through a collection called a device pool. During execution, tests will be run against all devices in the specified pool that is compatible with your application and tests.

Are any apps pre-installed on Testlio Automation test devices?

Yes, test devices will have a number of apps pre-installed by the device manufacturer or carrier.

Are devices able to communicate with other services or systems that are available on the Internet?

Yes. All devices have a WiFi connection with Internet access. If your systems are internal (that is, behind a corporate firewall).

Note: These IP and IP ranges are stable but not permanent. If you are unable to access Device Farm with these IP and IP ranges, then they might have changed since the most recent update of documentation.

Can I test different carrier connections and conditions?

While you can't test actual carrier connections, you can simulate connection types and conditions using the network shaping functionality. When scheduling a run, you can select a curated network profile like "Full" or "Half" or you can create your own, controlling parameters like throughput, jitter, and loss. All WiFi traffic from the device will be shaped and manipulated for the duration of your tests according to the profile you choose. You can also simulate dynamic environments by changing network parameters from your test scripts. For more details please see the specifying device states section in our automated run setup article.

Can I make phone calls or send SMS from the devices?

No, devices do not have carrier connections and cannot make phone calls or send SMS messages.

Can I use the device camera?

Yes, you can use the device cameras, both front- and rear-facing. Due to the way the devices are mounted, images and videos may look dark and blurry.

Which automation frameworks are supported?

Testlio Automation supports native and hybrid Android, iOS, and web apps, and cross-platform apps including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks.

For testing iOS, Android, we currently support Appium Java JUnit, Appium Java TestNG, Appium Python. For more information and an updated list, please visit our documentation.

How does Testlio Automation decide when to take a screenshot during a test?

If you use one of the supported automation frameworks, you are in full control and can decide when to take screenshots. Those screenshots are included in your reports automatically.

Does Testlio Automation support record and playback automation or do I have to write my scripts?

Testlio Automation supports frameworks like Espresso and Robotium that have record and playback scripting tools.

My app serves ads. Will they be displayed on your devices? Will my ad provider flag this as abuse and ban my account?

Although devices have access to the Internet, we make no guarantee that ads will be displayed. We recommend that you remove ads from the builds tested on Testlio Automation.

Can I access the machine hosting the device or access its shell as part of my tests? Can I reach the Internet from it?

Yes. If you’re using a client-server framework like Appium you can access the Internet and execute limited shell commands from the host.

I’d like to supply media or other data for my app to consume. How do I do that?

You can provide a .zip archive up to 4 GB in size. On Android, it will be extracted to the root of external memory; on iOS, to your app’s sandbox. For Android expansion files (OBB), we will automatically place the file into the location appropriate to the OS version.

My app makes use of location services. Can I specify the physical location of the device?

Yes, you can supply latitude and longitude coordinates that will be used to override a device’s GPS.

Can I run localization tests? How do I change the language of the device?

Yes, you can provide a locale (for example, “en_US”) to override the default locale setting on a device. Find out more in our how to create and automated run article.

How long does it take before my test starts?

Tests are immediately queued for execution and usually start within minutes. If one or more devices are not available, test execution for those devices will remain queued until they become available. Testing on the other devices in your test run will continue.

iOS: Do I need to add your UDIDs to my provisioning profile?

No, Testlio Automation will automatically replace a provisioning profile and resign your app so it can be deployed on our devices.

iOS: My app does not contain debug symbols. Can I supply a dSYM file to Testlio Automation?

No, but you can download the logs and symbolicate the stack traces locally.

Android: My app is obfuscated. Can I still test my app on Testlio Automation?

Yes, if you use ProGuard. If you use DexGuard with anti-piracy measures, we are unable to re-sign the app and run tests against it.

Android: Is Google Play Services available on your devices? Which version is installed?

Yes, Google Play Services is installed on devices that support it. The services are updated as new versions become available.

Android: Is there a default Google account on the devices?

No, devices do not have an active Google account.

Q: What’s in a Testlio Automation test report?

Testlio Automation test reports contain pass/fail information, crash reports, test logs, device logs, screenshots, and videos. Reports include both detailed per-device data and high-level results like the number of occurrences of a given error. Please check the understanding test results help article.

Q: Which device logs are included in a Testlio Automation report?

Testlio Automation reports include complete logcat (Android) and device logs (iOS), as well as logs from the device host and specified test framework.

For Mobile App testing:

For Desktop testing:

Q: What is Selenium?

Selenium is an open-source framework that automates web browser interaction. You can learn more about Selenium here.

Q: What is Desktop Browser Testing on Testlio Automation?

Testlio Device Farm enables you to execute your Selenium tests on different desktop browsers and browser versions. Device Farm follows a client-side execution model for Selenium testing i.e., your tests execute on your own local machine but will interact with browsers hosted on Testlio Automation through the Selenium API.

Q: How do I get started with Desktop Browser Testing on Testlio Automation?

To get started, please see our getting started guide.

Q: What operating system are the browsers hosted on?

All browsers are hosted on EC2 Windows instances which run on Microsoft Windows Server.

Q: What artifacts are available for troubleshooting test failures?

Device Farm generates console logs, web driver logs, action logs, and video recordings of the entire test to help you troubleshoot test failures.

Q: What desktop browsers does Testlio Automation support?

You can see here the list of desktop browsers and browser versions supported.

Q: Can I use Testlio Automation to test my web app on real mobile devices?

Yes. Device Farm supports testing web apps on real mobile devices using Appium.