Overview
Signal-driven testing is focused on detecting quality signals from various data sources and actioning them quickly and concisely to resolve quality that matters to application end users. Signals module incorporates event sources, detection rules, mapped actions, notifications and keeping track of signal lifecycle.
Setting up signals
Prerequisite
App Reviews must be set up first as source for signal generation as Signals currently only support App Reviews. We will introduce more sources in 2024.
User has a role Admin or Coordinator to set up signals
Add sources
Sources are hard requirement and connected to rules. In order to create rules, we first need to establish the source of information we want to keep scanning.
Go to Settings and select Sources
Click on "Add source" button
Select a source (we only support App Store and Google Play Store at this phase)
Select the available app from the source (prerequisite needs to be completed). Example if we have Netflix Inc app setup for App Reviews.
Add as source
Source now is listed as available source to connect to rules
Create rules
Rules are important part of Signals. Rules are needed for scanning sources and if we have a match, we generate signals. Consider it like monitoring rule that triggers and creates an alert if an event matches our search and scan criteria.
Important
Rules take effect on only future events, not profiling events backwards
App Reviews are imported daily, once a day at 5am CT
Rules are scanning once new reviews are imported
Profiling happens only to new app reviews from the import
If rules are modified, paused and put to live, then it applies to next import content
Setting up rules
Create a new rule in Signals > Rules
Define rule matching criteria
Rule name – provide a meaningful and identifiable name
Data source – select defined sources in previous step. Rule is linked to source and there can only be one source per rule. Important: you can create many separate rules per source, but you cannot share rule across sources.
Filters
Ratings – select one or many app review ratings that apply to the rule
Search term – comma separate keywords that need to be matched from the app review given by users. Each keyword is standalone condition
Signal creation behaviour
One signal per rule – option to sum all matching events into single signal. Example, when signal is scanned for new reviews and 3 matches, those are reported as 1 signal and all 3 reviews are shown as signal source events
One signal per event – each matching event is considered standalone signal and contains 1-1 mapping to the review.
Example rule setting
Created signal now is available in Rules listing.
Note – all newly created rules are by default in draft mode.
Edit, set live, delete and pause a rule
Select rule created in Signals > Rules
Rule detail view is opened
Edit – rule editing flow is opened and all defined settings can be modified. All edits affect only future events.
Set live – to active rule based event monitoring and scanning, rule must be set to live. Once it is live, we can see "LIVE" tag and Pause button
Pause – instead of deleting rules, we can pause them if the rule is needed or other needs tweaking, reviews by others. Once paused, user can set to live any time.
Delete – if rule no longer effective or needed, it is wiser to remove it.
Important 1– putting rule to pause or deleting one, does not effect already created signals. These statuses only affect future source scanning
Configure Slack notifications
We can check signals in the module and what have been created or closed, then most efficient way is to configure Slack notifications that will inform newly created signals.
Go to Signals > Settings > Notifications
This redirects us to workspace Slack notification settings
Select Slack workspace and select slack channel we want signals notification to be sent to
Select Signals:new and this will inform all in the channel of new signals
Add connection and we can see now notification is set up
Using signals module
Module overivew
Signals is based on monitoring events and depend if we have any matches to new reviews or not. By default it may be empty as our rules are not effective or the app is performing really well and users just give praise.
In case we do have signals generated, we do see them based on Open or Closed via our filters
In the list view we can search or fitlers and go to see detailed view of a signal.
Signals actions
Signals offers currently 2 options - close or create an issue to action it to understand if the feedback is causing impact to the application and is this quality gap in testing strategy or there is an issue we should try to reproduce and suggest to address it.
Create an issue
Create an issue from the detail view opens Issue module and connection to the issue.
Creating the issue offers the title and connects to signal itself, however, it does not auto-fill the issue itself. Does expect the user to fill in relevant information.
The issue is now linked to the Signal and we can see the corresponding reference as part of our signal in detail view
Connect as many issues to signal as relevant.
Close signal
Once the signal is actioned, we can close it with relevant reasoning and offering data points for future analysis.
