Jerad Raines (Lead UX Designer, Co-Researcher)
Early Concept testing
Design Refinement
Conclusion
Decreasing
User Effort to Review Accountability Reports
Accountability partners using Covenant Eyes’ new report summary can quickly determine whether or not they should be concerned about their partner’s internet activity.
Who I Worked With
My Role
UX Researcher (Primary)
Research METHODOLOGY
A/B Concept Testing
Quantitative Survey
Tools Used
UserTesting
Alchemer
Miro
Sketch
Confluence
Microsoft Excel
SPSS
The Company
Covenant Eyes
Covenant Eyes is a SASS company in the business of changing customer's online behavior. A pioneer in artificial intelligence, Covenant Eyes has been using AI-driven technology to facilitate human-to-human accountability for the past 20 years.
The Challenge
As an Accountability Partner, Should I Be Concerned?
Part of Covenant Eyes's core service offering facilitates accountability between the customer and an accountability partner to decrease undesired internet activity. As part of the service, accountability partners receive high-level reports of the customer's internet consumption and activity.
From past research, we knew that accountability partners could not determine whether or not they should be concerned about the content listed in a report. They wanted clear answers to the questions "Should I be concerned, what should I be concerned about, and how do I talk about it?" in the report.
The purpose of this iterative, mixed-methods research was to test and validate solutions designed to solve this recurring pain point.
RESEARCH PROCESS
EARLY CONCEPT TESTING
Early Concept Testing
Which Concept is Most Effective?
To address this challenge, a Covenant Eyes' UX designer created the Report Summary Module, to be added to the top of each email report. This feature aimed to help accountability partners quickly and accurately assess whether or not he/she should be concerned about the customer's recent internet activity.
The goal of this research was to test the effectiveness of two Summary Module design concepts compared to the current email report format. We hypothesized that the Report Summary Module would enable accountability partners to evaluate activity reports with higher accuracy.
The UX designer and I collaboratively planned and executed the experiment on the UserTesting platform.
Each accountability partner who participated in the study engaged with 1 of the 3 concepts and assessed whether they should be concerned about the activity. All design concepts and controls tested during this study only showed non-concerning reports.
I compared and contrasted these behavioral and attitudinal data including:
- Accuracy of task completion
- User confidence in task accuracy
- Ease of use to complete task
- User feedback to better understand user preferences, pain points, etc.
Descriptive Statistics & Affinity Diagramming
Insights From Concept Testing
A total of 28 Covenant Eyes accountability partners participated across the three tests (i.e., design concept 1, design concept 2, control). The data were analyzed in the following steps:
- I began by analyzing the quantitative data with descriptive statistics to assess concept performance.
- I ran inferential statistics on the data to identify whether the quantitative insights were representative of the larger accountability partner population.
- Then, the UX designer and I collaborated to analyze the qualitative data using an affinity diagram in Miro to uncover deeper insights.
High Accuracy Across All Designs
We found that accountability partners interpreted report content accurately across all three concepts (Chi Square Test, p=0.7778).
Some users struggled to decide because of image size and blur levels.
No Differences in Ease of Use or Confidence
During each test, accountability partners were asked to rate (1) the design's ease of use and (2) their confidence in interpreting the report accurately.
Krusal-Wallis tests confirmed there wasn't a significant difference in either confidence (p=0.202) or ease of use (p=0.116) overall. A Krusal-Wallis test was selected because the data weren't normally distributed, there was 1 independent variable with multiple levels, the dependent variable was ordinal, and the data were independent.
The New Feature Provides Value
Despite participants being able to interpret all three reports easily, confidently, and accurately, participants did have positive things to say about the new potential feature. Participants mentioned:
- “It’s really nice that it tells you that it didn’t detect activity to be concerned about right at the start.”
- “The summary is nice because it gives you an overview of everything.”
Although the report summaries were well received, they were used differently across users. Some accountability partners fully trusted the summary module data, while others additionally reviewed the screenshots because they said the artificial intelligence had been incorrect in the past. These findings align with previous company research.
However, the majority of participants were interested in using a Report Summary Module (an additional 18% said they would use it once remediations were made), suggesting that a Report Summary Module feature could provide value to accountability partners reviewing Covenant Eye’s clean reports. In fact, 24% of participants who tested the new feature mentioned that the summary module was the most important feature in the email report.
The report summary idea was also further validated by participants in the control group (who never saw the report summary design concepts) with comments such as:
- “I would like something more clear if I should be concerned or not.”
- “It would of course be nice to have a list of summary or warning about what happened…The recap is not enough. It’d like it to say at the top of the screen that nothing is wrong.”
- “It would be nice to have a summary instead of looking at the screenshots. It stresses me out.”
DESIGN REFINEMENT
UX Design Sprint
Data-Informed Design
Concept testing helped identify minor content and design changes that the UX designer used to make the concepts more user-friendly. Remediation fell into three main categories:
- Content clarity and consolidation to better tell the story and reduce scrolling
- Reorder content to get the information they need as quickly as possible
- Design Changes
EVALUATION
Quantitative Survey to A/B Test
Validating & Extending the Solution
A similar study was designed, but in quantitative survey format, so the refined design concepts could be tested and validated with a larger group of Covenant Eyes' accountability partners to get buy-in from leadership.
Additionally, this study investigated whether the email report could still meet accountability partner needs if shortened. The existing email report included a lot of content, which created many problems and design limitations. There could be an opportunity to re-imagine how email reports and other communication channels (e.g., My Account) work together.
Statistics & Affinity Diagramming
Quantitative Insights to Inform Confident Business Decisions
A total of 1,295 current accountability partners completed the survey, with participants randomly assigned to test one of the two design concepts. Both concepts were identical, except that Design Concept 2 featured a significantly shorted Recap of Activity section. For detailed activity information, accountability partners would need to access My Account. Both concepts included a summary module at the top of the email report.
Similar to concept testing, data were analyzed using descriptive statistics and a 15,000 sticky note affinity diagram.
The Majority of Accountability Partners Drew a Conclusion from the Report Alone with High Accuracy and High Confidence
Similar to concept testing, accountability partners were asked to use an email report to determine whether or not to be concerned about a person's internet activity. If needed, they were also given the option to visit My Account to view more screenshots.
The graph shows that survey respondents' conclusions, using either design concept, were accurate ~90% of the time. There wasn’t a statistically significant difference among the groups for successful interpretation (p=0.985).
Design Concepts Were Easy to Use
Decision-making was easy and made with high confidence. Accountability partners who selected either a 4 or 5 Ease of Use Score (for either design) reported they:
- Found the report summary at the top helpful and put them at ease
- Relied on the summary to decide whether or not to be concerned
- Were able to quickly use the report for the following reasons:
- The summary module helped them conclude quickly
- They could dig deeper if needed
- It was simple to view and use
The Shortened Report was Just an Effective as the Full Report
Approximately 70% of survey respondents from each group only used the email report to decide whether or not to be concerned about the hero's activity. In fact, accountability partners who only looked at the email report to make a decision did so with sigificantly higher accuracy (~99%). These data suggest it could be feasible to shorten email reports.
Chi-Squared Test of Independence did not find an association between design and next behavior (p=0.225).
Z Test for Proportions found a significant difference between report viewing behavior and accuracy for each design.
Accountability Partners Want to Replace the Existing Email Report
The quantitative data statistically validated that accountability partners want to use an email report that includes a report summary.
Participants were interested in opting in because:
- The upfront clarity of information with the option to dig deeper
- The summary and its clear visual cues facilitated quick review and decision-making
- Visual design’s aesthetic appeal, simplicity, ease of understanding and use
In general, users who chose not to opt into either design did so because they: were having challenges with the services, thought the current report was fine, or wanted to see additional design changes made.
Chi Squared Test of Independence detected an association between design and choice to opt in (p=0.022). Using a post hoc test, Z Test for Proportions found significant differences between ‘Yes and Not Sure’ and ‘Yes and No Choices.’
CONCLUSION
Multiple studies confirmed that Covenant Eyes' accountability partners saw value in adding the Summary Module concept to their email reports, considering it one of the most important features of the report.
The summary module easily and effectively helps users quickly evaluate a report and draw an accurate conclusion. The summary was a time saver for the full spectrum of report users. For example, we found that some users relied 100% on the summary itself to decide whether or not to be concerned, while others reported first reviewing the summary and then briefly skimming the screenshots to double check - decreasing the amount of time they typically spend carefully reviewing screenshots.
Findings from the follow-up quantitative study also suggest that that aren't negative consequences for shortening the email reports. There is opportunity to explore the relationship between the email report and additional communication channels.
Broader Implications
Insights from these research studies were used to develop a pitch for building the Report Summary module. Although the pitch did not move forward at the time of submission, the Summary Module was later built into Covenant Eyes' new Victory App, designed exclusively for accountability partners.
In Victory, the Report Summary Module played a critical role in summarizing the status of in-time internet activity feedback. It helped app users provide accountability more effectively.
The Report Summary Module continues to be iterated upon to provide greater value to accountability partners.