How to Guide Task-oriented Chatbot Users, and When?
My Roles
UX Researcher
UX Researcher
Company Info
National Yang Ming Chiao Tung University
National Yang Ming Chiao Tung University
Project Attribute
UX Research
UX Research
Mixed-Method Research
Best Paper Honorable mention Award
Timeline
Sept. 2020 - Sept. 2021
Sept. 2020 - Sept. 2021
Project Overview
Project Overview
Research Questions
Research Questions
Methodology
Methodology
Findings & Insights
Findings & Insights
Influence on Decision
Influence on Decision
Impact & Outcomes
Impact & Outcomes
Reflection
Reflection
OVERVIEW
When your chatbot breaks down, the guidance you show — and when — has measurable effects on task success, dropout, and whether users get better over time. This study maps the trade-offs across 8 combinations so your team doesn't have to guess.
Every task-oriented chatbot — booking assistants, internal IT helpers, customer service bots — shares the same failure mode: users say something the bot doesn't understand, and neither party knows what to do next. These conversation breakdowns kill efficiency, erode trust, and often end in abandonment.
The standard fix is "add onboarding guidance." But what kind? And when? Prior research gave vague recommendations ("show users how to interact with the system") without telling practitioners whether to use examples or rules, or whether to front-load guidance vs. surface it on demand. This study runs the actual experiment
When your chatbot breaks down, the guidance you show — and when — has measurable effects on task success, dropout, and whether users get better over time. This study maps the trade-offs across 8 combinations so your team doesn't have to guess.
Every task-oriented chatbot — booking assistants, internal IT helpers, customer service bots — shares the same failure mode: users say something the bot doesn't understand, and neither party knows what to do next. These conversation breakdowns kill efficiency, erode trust, and often end in abandonment.
The standard fix is "add onboarding guidance." But what kind? And when? Prior research gave vague recommendations ("show users how to interact with the system") without telling practitioners whether to use examples or rules, or whether to front-load guidance vs. surface it on demand. This study runs the actual experiment
RESEARCH QUESTION
RQ1. Which combination of guidance type and timing enables users to
a)complete their tasks more efficiently,
b)make better conversational progress, and
c)improve their performance during subsequent chatbot use?
RQ2. What are users’ subjective experiences of each of these combinations?
RQ3. What are users’ desired characteristics for the combination of a chatbot-conversation guidance type and its timing?
RQ1. Which combination of guidance type and timing enables users to
a)complete their tasks more efficiently,
b)make better conversational progress, and
c)improve their performance during subsequent chatbot use?
RQ2. What are users’ subjective experiences of each of these combinations?
RQ3. What are users’ desired characteristics for the combination of a chatbot-conversation guidance type and its timing?
METHODS
Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session
This case study compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience.
126 Participants (14 participantsx 9 conditions)
Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session
This case study compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience.
126 Participants (14 participantsx 9 conditions)
INSIGHTS SUMMARY
The key takeaways:
Examples warranted a good start, whereas rules promoted understanding.
The guidance timing matters.
The choices of both guidance type and timing depends on the chatbot’s application and the purpose of the guidance.
The key takeaways:
Examples warranted a good start, whereas rules promoted understanding.
The guidance timing matters.
The choices of both guidance type and timing depends on the chatbot’s application and the purpose of the guidance.
INFLUENCE ON DECISION
The findings don't give you a single "winning" pattern — they give you a decision framework. The right combination depends on whether you're optimizing for immediate task success, long-term chatbot fluency, or user satisfaction. Those goals often pull in different directions.
Statics Result
Scenario 1: Fast ramp-up + high satisfaction
Default to TASK-EXMP.
Show a copy-pasteable example at the start of each distinct task type — matched to the exact task, not a generic onboarding example. Place it adjacent to the input field so users don't have to scroll. Explicitly say "You can adapt this example.
Scenario 2: Users who self-serve over time
Use TASK-RL or REQ-EXMP.
Rules build transferable mental models. On-demand examples work for users willing to explore — they learn more because they've already tried on their own before reaching for help. Pair rules with an example: rules first, then "here's what that looks like."
Scenario 3: Error recovery without friction spikes
If deploying failure-triggered guidance, always use rules, never examples.
But reframe rules as "here's what to check" — not error messages. Give users a one-tap option to proceed with a suggested rephrasing. Tone matters as much as content here.
Never show an example after failure — users interpret it as "the right answer was obvious and you should've known." It generates resentment, not learning.
Scenario 4: Onboarding-only guidance
ONB-EXMP should be off the table.
It's the only condition that made users perform worse over time. Users skim it as visual noise ("looks like spam"), then can't find it when they need it. If you must show something at onboarding, use at most 1–2 generalizable rules — and link to task-specific help from within each task flow.
The findings don't give you a single "winning" pattern — they give you a decision framework. The right combination depends on whether you're optimizing for immediate task success, long-term chatbot fluency, or user satisfaction. Those goals often pull in different directions.
Statics Result
Scenario 1: Fast ramp-up + high satisfaction
Default to TASK-EXMP.
Show a copy-pasteable example at the start of each distinct task type — matched to the exact task, not a generic onboarding example. Place it adjacent to the input field so users don't have to scroll. Explicitly say "You can adapt this example.
Scenario 2: Users who self-serve over time
Use TASK-RL or REQ-EXMP.
Rules build transferable mental models. On-demand examples work for users willing to explore — they learn more because they've already tried on their own before reaching for help. Pair rules with an example: rules first, then "here's what that looks like."
Scenario 3: Error recovery without friction spikes
If deploying failure-triggered guidance, always use rules, never examples.
But reframe rules as "here's what to check" — not error messages. Give users a one-tap option to proceed with a suggested rephrasing. Tone matters as much as content here.
Never show an example after failure — users interpret it as "the right answer was obvious and you should've known." It generates resentment, not learning.
Scenario 4: Onboarding-only guidance
ONB-EXMP should be off the table.
It's the only condition that made users perform worse over time. Users skim it as visual noise ("looks like spam"), then can't find it when they need it. If you must show something at onboarding, use at most 1–2 generalizable rules — and link to task-specific help from within each task flow.
IMPACT & OUTCOMES
This was the first study to empirically test guidance type × timing combinations on chatbot task outcomes. Before it, practitioners were making real product decisions — onboarding flows, Help button placement, error recovery design — with no comparative evidence.
For UX researchers
A replicable methodology for evaluating onboarding guidance in any task-oriented conversational interface. The two-trial design (same guidance, different tasks) cleanly separates initial performance from learning — worth adopting in chatbot and voice interface studies.
For product designers
The performance-satisfaction divergence is the core actionable finding. Teams running A/B tests on chatbot guidance need to measure both completion rate and satisfaction — optimizing for one can actively hurt the other.
For content designers
The tone around failure-triggered guidance matters as much as the content. FAIL-RL is the most efficient condition but triggers the most negative emotional responses. Framing rules as "reminders" vs. "error messages" is a content intervention with measurable behavioral impact.
For growth / retention teams
ONB-EXMP is the most common real-world pattern (front-load examples at signup) — and it's the worst for learning. If retention depends on users building chatbot fluency over time, TASK-level guidance pays off more than elaborate onboarding flows.
This was the first study to empirically test guidance type × timing combinations on chatbot task outcomes. Before it, practitioners were making real product decisions — onboarding flows, Help button placement, error recovery design — with no comparative evidence.
For UX researchers
A replicable methodology for evaluating onboarding guidance in any task-oriented conversational interface. The two-trial design (same guidance, different tasks) cleanly separates initial performance from learning — worth adopting in chatbot and voice interface studies.
For product designers
The performance-satisfaction divergence is the core actionable finding. Teams running A/B tests on chatbot guidance need to measure both completion rate and satisfaction — optimizing for one can actively hurt the other.
For content designers
The tone around failure-triggered guidance matters as much as the content. FAIL-RL is the most efficient condition but triggers the most negative emotional responses. Framing rules as "reminders" vs. "error messages" is a content intervention with measurable behavioral impact.
For growth / retention teams
ONB-EXMP is the most common real-world pattern (front-load examples at signup) — and it's the worst for learning. If retention depends on users building chatbot fluency over time, TASK-level guidance pays off more than elaborate onboarding flows.
Reflection
Before conducting this study, I worked at Trend Micro on the design of a Hybrid Support system for intelligent customer service, where I was directly involved in product decisions around onboarding flows, guidance strategies, and error recovery in chatbot-human collaboration. This experience highlighted how, in practice, chatbot guidance design is often driven by intuition and isolated A/B tests, rather than a systematic understanding of how guidance type and timing interact.
Against this backdrop, this study reframes what is typically a fragmented and trial-and-error-driven design space into a structured, testable, and replicable comparison framework. From a business perspective, this contributes to significantly reducing the cost of experimentation in early-stage product iteration, minimizing the time and resources spent on repeatedly testing different onboarding and guidance strategies, and enabling teams to converge more quickly on design decisions grounded in empirical evidence.
Before conducting this study, I worked at Trend Micro on the design of a Hybrid Support system for intelligent customer service, where I was directly involved in product decisions around onboarding flows, guidance strategies, and error recovery in chatbot-human collaboration. This experience highlighted how, in practice, chatbot guidance design is often driven by intuition and isolated A/B tests, rather than a systematic understanding of how guidance type and timing interact.
Against this backdrop, this study reframes what is typically a fragmented and trial-and-error-driven design space into a structured, testable, and replicable comparison framework. From a business perspective, this contributes to significantly reducing the cost of experimentation in early-stage product iteration, minimizing the time and resources spent on repeatedly testing different onboarding and guidance strategies, and enabling teams to converge more quickly on design decisions grounded in empirical evidence.
Outline