Prioritizing Agentic AI Features
Determining how the business should prioritize the development of agentic AI capabilities according to user behaviors discovered in card sorting activities and qualitative interviews.
Research Question
How do we prioritize the development of Agentic AI tasks in a complex product with limited resources?
With several sub-products within a single product, each product team pushed its own tasks to be agentified first. Progress had stalled, and tensions between stakeholders were high.
Role: Research Lead
My job was to structure the decision-making process that would shape the next 1-2 year product roadmap.
Approach: A 2-phase mixed methods study over 5-6 weeks to answer 2 main questions:
What tasks are users comfortable
using Agentic AI for?
-
Unmoderated Card Sort (n=60)
-
Users were prompted to sort 42 product tasks into their preferred completion method (Agentic AI, AI-Assisted, or Manually) in an ideal product.
Why did users designate tasks
as Agentic or Manual tasks?
-
Moderated Card Sort Interviews (n=14)
-
Uncovering the rationale behind user designations.
Key Insights
Users consistently evaluated every task along two major criteria.
AI is only trusted to act in an Agentic manner if it can routinely tackle the tasks users already expect it to handle.
Artifacts
A stack rank report of all 42 product tasks with clear connections to research data.
A reusable strategic task classification framework.
An interactive questionnaire that product owners could use to classify new features.
AI opportunity was framed uniformly across all tasks and features with no clear prioritization.
Shifted prioritization to user-desired use cases first and established a phased development strategy.
Research Impact
Before Research
After Research
Design considerations for AI features in this context were ungrounded and loose.
Identified key requirements for user adoption of AI features and defined them as gating factors for any AI expansion.
Limited understanding of how users evaluate when to use AI for product related tasks.
Introduced a decision framework that explains and extrapolates user behavior across all product task types.
AI-Assisted Processes
Python scripts and Claude Code skills I designed reduced session evaluation time by 50%.
Claude Code and Replit were used to create interactive deliverables for stakeholders.
Have Questions?
Full study available on request.
Reach out and say hello!