In most cases Testlio uses 2 types of task lists:
- Exploratory task lists where you have to fill out a text box describing your testing strategy
- Structured exploratory or Regression task lists where you can choose whether the step passed, failed or is not answerable.
(There can also be other task list forms, in which case the instructions are included in the descriptions.)
In order to make sure that your testing goes well and the client gets the best overview of the results, there are a couple of rules we'd like to draw your attention to.
1. Describing exploratory testing results
With exploratory tests, testers should independently explore specific areas of the app using their own judgment and creativity. Because user scenarios and testing strategies are not specified in the tests, testers must describe their work process and results in the text area provided.
- All scenarios and areas covered in a short, brief, well-structured form
- Any found issues
- Any important notes that emerged
Task: Posting and commenting exploratory > Admin adding posts
Add new post with text only = OK
Add new post with photos = OK
Try to add post without internet connection = Appropriate notification is shown;
Try to open added photos to the post = unable: #30066
Post > add photos > remove photos before posting = OK
Add photos > Hold on photo > Minimize app = It freezes after resuming: #30078;
Task: Exploratory testing of the whole app for the remainder of task list time
Profile > Photos > Delete Photos = #27450
Feed > New Activity > Add Photos = #27457
Profile > Edit Data = OK
Profile > Activities > Turn off WiFi > Scroll down Activities screen to upload more activities = #41152
Contacts > Follow = OK
Invite Friends = OK
Comments: Generally, the UI for posting and managing comments is easy to use. Could be a bit more responsive though.
2. Choosing "Pass", "Fail" or "Unable to test" in structured tests
In structured tests, such as regression or structure exploratory, you can choose one of three options: passed, failed or unable to test. Tester can leave feedback to a test case in any circumstance ("Unable to test", "Fail", "Pass").
2.1 Choose "Pass": when the functionality works properly or with minor issues
For example when:
- User can log in.
- User can log in, but there’s a misaligned button.
- (updated October 4, 2017; released September 1, 2017) Add minor issue(s). In case there’s a minor issue and the client is accepting low priority issues, submit the minor issue and provide the issue ID in the field. All issue-related information should be included only in issue.
- Add feedback about test case if you have ideas how to enhance it.
NB! Selecting “pass” does not mean that you just didn’t find bugs, but that the functionality is working as expected. Common mistakes is to select “Pass” when you can’t test (and because of that can’t find bugs either), but in that case Pass should be selected, minor issue added and/or problem explained via "Add feedback about test case"
2.2 Choose Fail: When the functionality is broken or actual results don’t match with described expected outcome.
For example when:
- User cannot login, throws blank screen.
- User reached login screen but the action cannot be successfully completed in some options - i.e Facebook login works, but email login does not.
When that happens submit the issue and provide issue ID in the field.
If the bug is from the past, please add reproducing comment inside the issue and report issue ID as regular.
- (updated October 4, 2017) Report the issue ID, for example #85589. All issue-related information should be included only in issue.
- (released latest September 1, 2017) Add feedback about test case if you have ideas how to enhance it.
2.3. Unable to test: choose in cases when the functionality cannot be tested or you feel the need to leave feedback.
- (Updated October 4, 2017) Blocked by another issue. There’s a bug that blocks testing i.e the video player crashes while loading, so it’s not possible to test play and pause functionality. If that happens link the blocking issue ID in the comments. All issue-related information should be included only in issue.
- Functionality out of scope
- Don't understand reason if test case description is hard to understand for testing.
- (Updated October 4, 2017) Other reason if the feedback does not match the criteria above, please specify the reason via comment. For example "Can't login because staging environment problems."
In addition, you can Add feedback about the case to help enhance the case.
NB! Commenting on the reasons why the functionality can’t be tested is important to highlight the parts of the application that could not be tested and to build trust.
If you're still unsure or in doubt regarding reporting a specific result, please ask in project chat.