What is a Testing Window?

From a tester’s point of view, a Testing Window is a time-slot where testing activities in a run occur. It has a clear start: the moment when the tasks become available for execution and we expect testers to start testing, and a clear end: the latest moment by which all testing tasks in a run have to be completed.

Why use Testing Windows?

Previously at Testlio, one customer request was often tested within a single long testing period - sometimes ranging from 12- 48 hours. We are re-shaping this approach so one customer request can be served by one or multiple testing windows which start sooner, have a shorter duration, and have predictable start and end times.

How do Testing Windows Work?

To handle a client testing request, a coordinator will schedule one or multiple testing slots with a duration of up to 4 hours. The schedule will be decided based on the coverage and turnaround time requirements accounting with availability of testers during particular times.

In short, it means that we expect testers who accept the invitations to work closely during a shorter period of time ensuring timely support, collaboration, and results delivery.

Why this change?

While our previous model provided great flexibility for testers to choose a suitable time within a long testing window duration to start the work, in practice, it has caused multiple issues for our Test Leads, testers, services team, and our customers. With the previous approach, testing was often started close to the end of the run. This meant that critical blockers that could have been signaled to the customers early, are found later.
​
Additionally, it’s been hard for our Test Leads to support our testers over long runs, sometimes encompassing entire weekends. This has led to frustration for testers when they do not receive timely support, amplified by how many teams operate around the world across time zones. Importantly, Test Leads experienced pressure due to the need to stay online for extended periods of time, even when testing was dormant.

Via extensive modeling and trialing new approaches through 3 rounds of pilots, we have come to the conclusion that shorter periods of testing time lead to increased collaboration, prompt support, better Test Lead experiences, and better results for our customers.

How does this change impact you as a tester?

First, you still have the same chances of receiving an invite to a run that you had previously. Our platform still considers coverage requirements and freelancers’ availability.
​

Second, as we roll-out new shorter testing window durations in our Workspaces, we expect you to show up and start testing within the first hour of the testing window and generally complete the testing by the beginning of the last hour of the window to make sure you have enough buffer time to complete activities, file issues, and respond to input. Our coordinators will do their best to ensure that every tester has the time to complete their task to account for unforeseen circumstances or extensive issue reporting time. In general, the testing window duration will be two times the duration of the longest testing task. While for now this approach is a recommendation, in the future we foresee making this mandatory, noting that we are working on additional Platform enhancements to support our vision.

Third, with the shorter run duration, we see the time management becoming more straightforward. Having clear time-slots within a day where you will be invited to test should allow you to focus on one task at a time (noting our Freelancer Services Agreement prohibits parallel work/double billing).

Additionally, you should receive more prompt responses to your questions from Test Leads—and you will generally have fellow testers doing test execution with you at the same time.

We encourage you to share your observations about the builds and products you are testing with others in the relevant Workspace chat to share experiences and ensure great results for the testing team and our customers.

When will the change happen?

We will be rolling out the concept change gradually. Some Workspaces are already operating this way as of March 2023. We will closely monitor the roll-out, actively seek feedback, and work to address any concerns. We intend to apply this approach throughout all of the applicable engagements within 2023. Our Testing Managers, Testing Coordinators, and Engagement Managers in each workspace will communicate upcoming changes to their teams to help avoid surprises.

FAQs

Will all tasks occur in a Testing Window?

Not all tasks will be limited to testing windows. Some engagements will require longer testing times or may not be suitable for the testing window approach. Many tasks associated with a run (build intake, instruction preparation, results analysis, client reporting, etc.) will intentionally happen outside of the window. However, we strive to conduct the majority of our planned and scheduled testing work within the testing windows in most Workspaces.

What is the recommended and maximum task duration within a window?
A recommended task duration within a window is up to 1h and 30 minutes. Maximum task length within a window is 2 hours.

Is it possible for me to participate only in the Testing Windows runs that are scheduled in my region?

You are free to choose whichever testing windows that work best for you, regardless of your location. However, we ask that you only accept invitations to testing windows when you can be productive.

Is it possible to accept invites to multiple runs that are happening within the same 4h testing window?

Today, our Platform does not prevent this. However, we request that you only accept invitations for testing work in a window that:

Per below, you also can accept invitations for other tasks in the window, like issue reproduction, once you complete testing.

Does the 4-hour testing window include issue reproduction tasks?

Issue reproduction tasks are additional tasks. We aim to start reproducing issues as soon as they are reported during the testing window. In general, these are invited and compensated separately, meaning you may have more work in the window than the initial scope of testing.

Will invitations that expire during my night time affect my rating?

Invites sent and expired between 11 PM to 7 AM in your local time zone will not affect your ratings negatively. Although you may still receive invites during this time, the expired ones within this 8-hour range will not be considered in the rating's calculation.

#TestlioBot