Scaling Research Across a Product Family
Building a research cycle to evaluate 50 usability study requests by creating a new AI-assisted research process and strategically selecting studies.
Research Question
How can I deliver research that covers as much as possible in a large quantity study request?
One successful usability study brought in a request to repeat the process 49 more times across the same product family. Because it is impractical to replicate a study that many times, I needed to find a way to provide efficient and effective research support.
Role: Research Lead
My job was to evaluate a product plug-in, which then evolved into providing research support over a broad product family.
Initial Approach:
A 4 week moderated usability test (n=8) evaluating a product plug-in that enables users to execute tasks in a host software outside of our own.
The product plug-in is comprised of 4 discrete features (A, B, C, D).
These features are independent of each other.
Usability Test Results
"This is so helpful!
Can you do this for our other plug-ins in 49 different host software?"
-Product Team
-
Features A and C were well-received and had no major usability concerns.
-
Feature B was effective, but users expressed varying preferences.
-
Feature D needed to be revised and had critical usability concerns.
The identified revisions were implemented and product launched successfully on schedule.
Re-Framing
50 studies at 4 weeks/study
=
200+ weeks of resources
I worked with my stakeholders to explore alternative options.
Feature composition, appearance, and location may vary, but features will always complete the same function.
We only had enough resources for a handful of testing environments and licenses for these studies.
Each plug-in feature (A,B,C,D) had variants (A1-A5, B1-B3, C1-C7, D1-D2) that were used throughout all the 50 plug-ins.
B1
B2
B3
The data the team needed was primarily around the 4 individual features (A,B,C,D) in each plug-in.
A
B
C
D
Study Selection
By mapping all the expressed plug-in feature variants to all 50 host software, I could identify which software would provide the most coverage.
6 selected studies would cover 98.5% of the feature variant instances across all 50 plug-ins.
Feature Variant Matrix
Coverage Matrix
AI-Assisted Research Cycle
Even with only 6 studies instead of 50, the request was still too large.
6 studies at 4 weeks/study
=
24+ weeks of resources
By switching to unmoderated methodology, introducing Claude Code Skills, and altering deliverables I cut a 4-week research cycle down to 2 weeks.
Results
With the reduction in time and resources, the new research plan was executed.
6 studies at 2 weeks/study
=
12+ weeks of resources
A searchable usability severity matrix for all the tested variants
A detailed report outlining contextual findings from the 6 host-software in the studies.
Artifacts
Research Impact
Before Research
After Research
Plug-in features would be launched without any research validation
All plug-in features had research they could reference prior to launch in a searchable codex.
Evaluating 50 plug-ins meant 50 studies
6 studies covered 98.5% of variants across the 50 host software plug-ins.
With 6 weeks to launch, the team did not know if the plug-in worked for users.
Ready features and revisions needed were identified. Fixes were shipped and the launch was on schedule.
Have Questions?
Full study available on request.
Reach out and say hello!