This article presents a requirement for our freelancers. Failure to follow the process might represent a breach of the Freelancer Services Agreement.
In this article, we will explain how to execute tests and how to mark up results in order to ensure a clear overview of results for our clients and a smooth testing experience for yourself.
Execution Flow
Before jumping straight to the test case execution, skim through the test steps to understand the length and purpose of the test. This helps you to understand the context and be successful in time management.
Bulk marking pass/fail results is not allowed. Make sure to execute a step at a time and mark up results as soon as the step gets executed - this helps you to ensure that you cover all details in a particular test step and provides us and the customer with context about the execution itself. Even if for some reason you will need to re-execute the step and the outcome will change, you can always update the results before the task submission. Please note that analytics workspaces will be considered as exceptions, and marking steps in bulk is allowed in workspaces where analytics testing is conducted. Moreover, when a Testing Manager or an Engagement Manager explicitly states the need for bulk-marking in a particular workspace, they should also provide an explanation to help testers understand the rationale behind this decision.
We expect you to focus on one task at a time. Even if you have multiple tasks in the queue, switching the context to another task without a valid reason (e.g. you being blocked in the first task) is highly discouraged. Any form of parallel task execution is against FSA and may result in off-boarding from the platform.
Marking Results
The most commonly used types of tests at Testlio are:
Exploratory tests – that article goes into details on how to mark results for exploratory tests.
Scripted test – this includes structured exploratory and regression tests.
During scripted test execution, the testers follow structured test steps and can choose whether a step passed, failed, or if the tester is unable to test the particular step at all.
Pass
Select this option when the functionality works properly. It is possible that you might still find some minor problems even though the main functionality is working normally. For example, if a test step requires the tester to log in to an app and the tester can:
log in
log in, but there is a misaligned button.
log in, but found some interface-related issues with forgot password flow.
In such situations, where the test steps pass but also have a minor problem, you need to report an issue and choose the severity as Low. Before creating a new issue, make sure that a duplicate issue does not exist, if it exists then you need to follow the issue reproduction process.
Fail
Select this option when the functionality is broken or actual results do not match fully with the described expected outcome in the test step. For example, if a test step requires the tester to log in using email, but the tester is:
unable to log in as it throws a blank screen
able to login with a Google account, however, email login does not work.
Once you click the Fail button, an issue browser will open where you can look for duplicate issues to reproduce or create a new issue.
Note: It is a common mistake to select Fail when the feature described by the test functions as described in the expected result but you find a related issue. In such cases, you should select Pass and then link that issue to the test case using the + Issue button.
Unable to Test
Select the Unable to Test (UTT) option when the functionality described in the test step cannot be tested. You can choose one of the following reasons why you were unable to test:
Blocked by another issue - Use this reason where there is an issue that blocks testing the test step completely
You’re asked to test the play and pause functionality but the video player crashes while loading, so it’s not possible to test play and pause functionality.
Make sure to link the blocking issue from the issue browser
All issue-related information should be included only in the issue
If the whole task list or majority of the tests are blocked
You need to also mention this in the workspace chat for the Test Lead as the Test Lead may have a workaround
Test Lead needs to notify the customer that we are blocked on testing critical area
Functionality out of scope
Don't understand - if the test description is hard to understand and you did not receive help from the Test Lead. Please note it’s not ok to mark it as ‘Don’t understand’ without talking with a Test Lead first.
Other - When the reason does not match any criteria above, please specify the reason via comment. For example, "Could not log in because of staging environment problems". Again, please make sure you first talk to the Test Lead to confirm that it is OK to mark this as UTT.
Please keep in mind:
Commenting on the reasons why the functionality can’t be tested is important to highlight the parts of the application that could not be tested and to build trust.
If the problem is in the test case, make sure to leave also test feedback.
Make sure to always critically evaluate if all the steps in the test are blocked - sometimes, you can still execute the test further (for instance, if a rewind button in a video player is now missing because of the design change, then other steps covering fast forward, play, stop and others can be still tested)
If you're still unsure or in doubt regarding reporting a specific result, please ask in the workspace chat and consult with the Test Lead.
Feedback
Testers can add feedback about each step to help enhance the test. It can be done regardless of what option (Pass, Fail, and Unable to Test) they have selected for the particular step.
Feel free to add feedback about the test step if:
The step seems outdated - the feature seems to work fine, however, the step states otherwise.
The step is hard to understand - add suggestions of what could be added in order to explain it better.
The order of the test steps could be improved - also suggest what could be changed.
#TestlioBot