This article presents a requirement for our freelancers. Failure to follow the process might represent a breach of the Freelancer Services Agreement.
The Freelancer Incident Reporting System's purpose is to provide visibility of the unacceptable actions of our freelancers and to maintain good quality within our network.
At Testlio, we recognize that mistakes can happen, and we believe in second chances in most cases of quality problems. However, there are certain incidents that can result in immediate off-boarding from our network.
As transparency is very important for us, we want to communicate unacceptable actions to everyone.
What happens when you are reported?
When a freelancer is reported, they will receive an email explaining the incident under investigation and asking for their perspective. The Testlio team will review the incident and assign the severity. Depending on the nature of the incident, the freelancer may receive direct feedback and guidelines for improvement, along with a second chance to improve their behavior. For freelancers with previous reports, we take into account the date of previous incidents when evaluating the decision on the next steps.
Points calculation and expiration
The reporting system is based on a points system that will determine when the freelancer is off-boarded and will set a clear flow for points expiration - thus putting in a human element of understanding that everyone can make mistakes from time to time.
Once the freelancer reaches 10 points, they will be off-boarded.
All points earned from incidents within the past year will be 100% considered in the analysis.
One year after your last incidents, 50% of your points will be deducted. After the second year, all remaining points are deducted.
Points from incidents last for two years. If no new incidents occur within those two years, all your points will expire, and you'll have 0 points.
To identify recurring patterns and assess a willingness to learn from past mistakes, all previous incident reports may be considered during incident analysis, if needed.
Incident Report Categorization
Each incident report will be categorized against one reason brought in the table below and the points will be assigned per categorization.
Reason | Description | Severity | Points |
Clarified non-impact | Low-impact scenarios have been clarified with the freelancer. Only the following low-severity incidents:
1. The tester did not include one of the attachments in their issue report, but they added it later when the coordinator asked for it.
2. The tester did not notify the coordinator about spending less time on testing but included it as feedback for the task.
3. The tester keeps reporting poor-quality issues and hasn't improved despite giving feedback. The team already talked about these problems with the tester, and they understood their mistakes. Note: This list can be updated if new instances emerge.
5. Quality Engineering specific - Inadequate coding efficiency. The code does not follow good language-specific practices, contains poorly written logic (e.g. not leveraging guard clauses), or contains grammatical/syntax mistakes. | Low | 3.5 |
Informed no-show | The tester missed the run, but they informed the test lead either before the run ended or within 24 hours after. They had a valid reason, such as a personal emergency or health issues. | Low | 3.5 |
Bulk markup | The tester marked the entire task or step groups as pass or fail in a very short time frame, either at the beginning or end of testing, due to a bad practice or habit. However, they didn't miss any issues in the task. | Low | 3.5 |
Rushed testing | The tester marked the entire task or step groups as pass or fail, and also missed an issue that other testers had reported or that had been present in previous runs.
| Medium if driver is: Insufficient Testing or Misaligned Testing
High if driver is: Misrepresented Testing | 5
10 |
Invalid pass | There is clear evidence that the tester missed an issue as they marked a step as "pass" despite the feature not functioning properly. | Medium if driver is: Insufficient Testing or Misaligned Testing High if driver is: Misrepresented Testing | 5
10
|
Inaccurate time reporting | The tester did not work (test) the full amount of time they were allocated and paid for, and they did not accurately report actual lower time worked. | Low if the driver is Insufficient Testing or Misaligned Testing and the time difference is small Medium if driver is: Insufficient Testing or Misaligned Testing High if driver is: Misrepresented Testing
| 3.5
5
10
|
Device mismatch | The tester conducted testing with a different device than the allocated one. | Medium if driver is: Insufficient Testing or Misaligned Testing High if driver is: Misrepresented Testing
| 5
10
|
Configuration mismatch | The tester used the wrong build/environment/configuration (including accounts, etc) for testing/engineeringtasks. | Medium if driver is: Insufficient Testing or Misaligned Testing High if driver is: Misrepresented Testing
| 5
10
|
Unjustified no show | The tester did not show up for the run or did not complete the task and failed to communicate this to the team. This includes situations where the tester hadn't even started the task, and their estimated task execution time would have exceeded the run end time. | Medium if it is the first occurrence High if this happened previously or if there is no reply and the tester is not active.
| 5
10
|
Instructions non-adherence |
The freelancer did not follow the run/task instructions/proposed guidelines. | Low if driver is Insufficient Testing: Medium if driver is: Misaligned Testing High if driver is: Misrepresented Testing
| 3.5
5
10
|
FSA disobedience | The freelancer violated any of the following high severity Freelancer Services Agreement (FSA) points:
| High (Misrepresented Testing) | 10 |
Other | Any other case that cannot be categorized to the existing pre-defined reasons | Low/Medium/High (it will be analyzed case by case) | 3.5/5/10 |
For several of the reporting reasons, there will be 3 different drivers based on Reports Categorization Taxonomy:
Insufficient Testing (Unintentional missed issue. Freelancer did the work by following instructions, but missed the issue, potentially because of fatigue, misunderstanding, miscommunication, limited time, or other problems.)
Misaligned Testing (Freelancer did not follow the instructions given in the task potentially due to lack of attention, sloppiness or other factors, resulting in a wrong testing setup or incorrect testing scenarios.)
Misrepresented Testing (Intentional breach of the Freelancer Services Agreement. Freelancer knowingly misrepresented their work (e.g. false passes), location, equipment, skills, time, or other factors (can include freelancer subcontracting work to someone else and/or parallel testing).)
Severity definitions
Low - Although the workspace and client were minimally affected, it remains essential to report and document the incident to track it in case of a similar situation in the future.
Medium - Unintentional mistake from the freelancer’s side.
High - Intentional misrepresentation of their work.
Useful information
Please refer to the article: Avoid Getting Reported
#TestlioBot