Aaron Stites, UX Researcher
Nate Schloesser, UX Team Manager
Covenant Eyes Stakeholders
Increasing
UX Research Impact at Covenant Eyes
Making research insights centralized and accessible to meet stakeholder needs.
Who I Worked With
My Role
Lead UX Researcher
Project Manager
Research Methods
Content Audit
Remote Focus Groups
Competitive Analysis
Remote Interviews
Tools Used
Miro
Zoom
Otter.ai
Confluence
Outlook Calendar
Condens
The Challenge
Qualitative Data & Findings Scattered and Under Utilized
Shortly after joining Covenant Eyes as a researcher, I discovered that UX research reports were hard to find and not centrally located. In fact, colleagues often mentioned that some research had been lost, duplicated, or was never accessed by stakeholders. Data were also scattered across tools, making it difficult for users to retrieve past data and identify trends across projects.
Consequently, I initiated and conducted internal research to understand how research findings were used by stakeholders, identify pain points, needs, etc. My goal for this project was to increase the findability, usability, and awareness of Covenant Eyes research projects and their outcomes, which in turn would help increase UX research's impact.
Design Process
Discovery
Content Audit
How is Company Documentation Organized?
UX research documentation was a disorganized mess, but I wondered what could be learned from other departments within the company. How were they organizing and sharing information?
I learned that departments used the company-wide documentation tool, Confluence. Like UX, each department's Confluence space was text-heavy and a dumping ground for poorly organized, hard-to-find content.
To better understand stakeholders' experiences with UX research findings, I led 4 remote focus groups (which included several interactive activities), with Covenant Eyes stakeholders who use UX research. My goal was to identify their goals, needs, and pain points using UX research.
Participants were 20 Covenant Eyes employees from across Product, Marketing, Tech, and Business Intelligence who fit into the following categories:
- Individuals who Rely on Research to Make Decisions
- Partners Who Help Support Our Research
- Executives
Follow-up 1-on-1 interviews were conducted with several participants to expand upon topics discussed during the focus group(s).
Themes From User Data
Stakeholders Seek Out Research With a Specific Purpose or Question in Mind
As a result of the focus groups, I learned that colleagues in Product and Marketing normally seek out research for a specific reason or purpose (e.g., to answer a question, find out what we already know about a specific topic, and/or understand a problem). This is especially true when they begin a new project.
Stakeholders Typically Engage with UX Research in Two Ways
The data suggested there are two primary ways users consume UX research at Covenant Eyes:
- Some stakeholders want to quickly and easily get the gist of the findings with the opportunity to dig deeper if needed.
- Other stakeholders want to dig into the raw data to solve new problems.
There is a high interest and potential value in using our pre-existing raw data to answer new research questions. However, the raw data wasn’t available in a centralized location or easily searchable.
Researchers Company Wide Experience Similar Challenges
During the focus group discussions with Covenant Eyes Marketing and Business Intelligence teams, we learned that although each team uses data differently, we all have been experiencing similar challenges such as:
- We all have challenges with data and we’re independently working on solutions for our data to be more easily consumed.
- None of our teams have a formal research repository. Our data is scattered across tools.
- No standardized naming conventions and taxonomy are used.
- We have experimented with tools.
Stakeholders Cannot Find Research Without Asking a Researcher for Help
People who regularly use UX research cannot find the research in Confluence. They reported that research is scattered across Confluence and they don’t know where to look. Consequently, many of the focus group participants said they skip looking through Confluence and go directly to the researcher to find research.
There is Interest in Big Picture Trends and Using Data to Positively Impact the Company
One theme that emerged from the data is that people who use research to make decisions (e.g., UX designers, product leaders, marketing) are interested in finding connections and trends across research projects and data collected. A current problem recognized by UX researchers is that UX research findings and data are is currently siloed within each individual project.
There is hope that a research repository will positively impact the company: impacting decision making, connecting users with stakeholders, informing research projects, showing transparency of research, preventing unnecessarily research replication, etc.
Note: Participants were each given 5 stars to vote on what features mattered the most to them. (1) Seeing trends (shown above) and (2) research’s potential impact on decision making received the most votes.
Ideation
Research Repository
What Functionality Do we need?
From the user research, we found that Covenant Eyes needed a research repository that could:
- Centralize and organize qualitative insights and data
- Support quick data analysis and reporting
- Provide a participant database
- Identify data trends and connections across projects
- Automate and streamline research processes
- Make UX research accessible, approachable, findable, and easy to use
- Consistently showcase, summarize and store UX research in a standardized way
- Provide stakeholders self-service access to effortlessly retrieve both research reports and pre-existing raw data (that can be filtered and searchable) to help guide their decision making, answer new research questions, and show what we have been working on
- Be secure
- Increase stakeholder engagement with UX research
We needed to decide whether to adopt an internal or external tool for the research repository. Since the research groups were experiencing and trying to solve similar problems across the company, it was possible, that we could solve this problem collaboratively with the same solution.
During the focus groups, many of the participants mentioned their experiences using Confluence. Table 1 shows the disadvantages outweigh the advantages of using Confluence for our repository.
I and our other UX researcher also met with each research group and the leader of our company's tech team to learn about their current tools (e.g., Oracle, Hubspot) and whether they could be an effective solution for this project.
We learned:
- none of our pre-existing tools could provide all the essential functionality we needed.
- Building an internal tool would be time-consuming and not cost-effective.
Table 1. Confluence Pros and Cons Identified by Participants
| Confluence Pros | Confluence Cons |
|---|---|
| It's the company standard | Some stakeholders don't use Confluence or stay in their own space |
| Everyone has access | Hard to find content |
| It has a JIRA integration | Content deleted without authorization |
| Search functionality | Some participants have a mistrust in its search functionality |
| Analytics plug-in | Everyone's space within Confluence looks different |
Competitive Analysis
Exploring Potential Solutions
I conducted a thorough competitive analysis to investigate and compare 15+ UX repository tools (across 75 different criteria) currently sold on the market. Data were collected from vendor websites, help documentation, and sales meetings. Based on our needs, we narrowed down our selection to three potential tools: EnjoyHQ, Dovetail, or Condens.
I and our other full-time UX researcher conducted free test trials with each of these three tools to better inform our selection. A brief summary of each tool's pros and cons is listed below.
EnjoyHQ
$1,000Per Month
Dovetail
$900Per Month
Recommended
Condens
$135Per Month
Recommendation: Purchase Condens
Based on our team's needs, budget, competitive analysis data, and test trial experiences, I recommended (and our other UX researcher was in agreement) that we move forward with the purchase of Condens.
For our UX team, not only was Condens the most cost-effective option (approximately a $10,000 annual savings compared to other prospective tools), but it also best met our needs with its quick analysis and report building functionality, unlimited participant database, user-friendly interface, global tagging functionality, and dedicated stakeholder area. Stakeholder feedback about Condens (and reports created in Condens) during the test trial was also very positive.
Design & Testing
Setting Up the Repository
Selecting and purchasing Condens was honestly just the beginning of this project. Besides Condens basic infrastructure, your repository begins as a blank slate. For example, we needed to figure out our tagging and metadata content and infrastructure, whether to add old data and reports, integrations, privacy, etc. We also needed to design the repository for efficient, streamlined metadata and data entry for researchers to ensure the completeness and continuation of the repository itself.
Remediating a research repository such as this would be a costly, time-intensive task (note: I know because I'd been hired to remediate one in the past), so I wanted to make sure we set up the repository for success from its inception.
Metadata & Content Strategy
Designing for the Content Users Need
One of our goals for this project was to increase the use of our research findings. For stakeholders to find the research repository useful, I wanted to make sure the information they want and need would be available. Table 2 shows the content needs that emerged from the focus group research.
Table 2. Stakeholder Research Content & Metadata Wishlist
| Report Preview Metadata | Content Needed in a Research Report | Content Organization |
|---|---|---|
| Date research conducted | Executive summary | Projects categorized |
| Abstract | Who conducted the research | Connects with JIRA help desk |
| Project has a meaningful name | Participant Demographics | Shareable |
| Tags | Methodology and test design | Browse and search functinality |
| Concepts tested | Links related projects | |
| Research findings & links to raw data | Makes connections across projects | |
| Learning cards | Consistent taxonomy and naming nomenclature used |
Global & Project Tags Setup
Which Themes Should We Track?
Covenant Eyes UX research had been siloed across multiple tools and projects. Although I had noticed recurring themes across our data sets, there hadn't been an efficient way to aggregate and quantify the data. The global tag functionality (i.e., use the same tags across projects) in Condens would solve this problem!
My UX research colleague, Aaron Stites, had previously developed a tagging system (which included more than 100 tags relevant to the company's service) for several of his recurring surveys. These same tags would be very applicable to many other projects, so they were selected to be the foundation of our global tagging infrastructure in Condens. This tagging system was entered into Condens and continues to be expanded upon as needed.
Project-specific tags will be added on a project-by-project basis as project tags (i.e., tags that can only be used within a particular project).
Information Architecture of Metadata
Streamlining Metadata Entry & Searchability
The biggest challenge in designing the research repository was creating an information architecture for the metadata. In Condens, metadata exists at 3 levels (i.e, project, research session, and participant profile) and inheritance rules apply. This means that research sessions inherit project-level metadata, and participant metadata inherits both project and the participant's research session data.
We knew from research that our stakeholders wanted a self-service, dedicated space for research, and which metadata they needed. I wanted to make sure that the findings and data nuggets would be easy to search, use, and find using metadata, while also making data entry quick, easy, and non-redundant for researchers.
With advice from Condens experts, data from stakeholder interviews, and usability testing, I was able to assign, streamline, and optimize metadata for each of the 3 levels (shown below).
Project Metadata
Research Session Metadata
Participant Metadata
Developing Training & Resources
Onboarding the UX Team into Condens
To align our team on the tool, establish consistency of use and teach best practices, I developed a remote training session and reference materials.
Conclusion
Our research repository is live! Moving forward, all our UX research data and documentation will reside in our research repository.
Using Condens has been a huge time saver for writing our research reports. For example, a report that typically took me 2-3 full days to write and format in Confluence now only takes me ~3-5 hours to analyze the data, create video clips, and write a full report. That's approximately 19 hours of time savings!
Now our research is accessible to everyone at the company. We've found that Condens has been very well received by stakeholders. For example, with a smile on his face, our project manager told us, "this report is perfect!" Our UX designers are also eager to use the tagged raw data for upcoming projects.
Although the data tagging is still in its infancy, it will continue to offer more and more value as we add more tagged data into the repository.
Next Steps
Moving forward, one of my roles is to maintain and oversee the repository to ensure consistency and completeness of data entry. I also provide technical support and tips as needed.
This project is a stepping stone for future research projects. With Condens, now we can quickly and easily reuse pre-existing data to answer new research questions (when applicable) and conduct high-level analysis across our full dataset to identify and quantify critical themes and issues.
Focus group data also revealed that there is strong interest across departments for updates about what UX research is working on, has completed, etc. Although Condens provides a space for people to learn about recent UX research projects, they must actively seek it out. My next focus will be developing content and communication strategies that will focus on disseminating research directly to stakeholders' preferred channel(s) of communication.