Tutor Co-Pilot
An LLM-powered Tool
For Math Tutoring
As a product design intern at PLUS, I designed an LLM-powered Co-Pilot to help tutors explain math concepts more effectively and enhance engagement during tutoring sessions.
ROLE
Product Designer
TEAM
2 Product Designers
1 Product Manager
1 Designer Manager
1 ML Engineer
1 Full-Stack Developer
DURATION
Feb - July 2024
(6 Months)
WHAT I DID
Product Scoping
Prompt Engineering
GPT Co-Design
Parallel Prototyping
Usability Testing
Model Iteration Doc
WHAT IS PLUS?
An AI-Driven Online Math Learning platform.
PLUS, where I interned as a product designer, is a math tutoring platform that connects human tutors with AI-powered software to enhance learning outcomes for middle school students from historically underserved communities. The platform supports over 3,000 students and 500 tutors, completing more than 90,000 hours of tutoring each month.
CONTEXT
Racing against the clock
At PLUS, tutors conduct 30-minute math tutoring sessions with groups of 5 students, giving each student roughly 6 minutes of individual attention. This limited time makes it challenging for tutors to fully address math problems, resulting in rushed explanations or going over time.
Under the age of AI
Following the wave of AI, the product team wants to design an LLM-powered solution to help tutors with this issue.
MY DESIGN
Conversational AI Tutor Co-Pilot.
I led the design of an LLM-powered Co-pilot to help tutors clearly explain math problems, suggest words of encouragement, and ask strategic questions, enhancing time-efficient and engaging tutoring sessions.
001
Start By Inputting A Math Problem
002
Polish the Model’s Output with Follow-up Suggestions
003
Instant Feedback System For Effective Model Iteration
IMPACT
Elevating tutoring experiences for all stakeholders!
100+
Monthly Active Users Among 500+ Tutors Across The Platform
FOR STUDENTS
+38%
Increase In Students' Session Engagement
FOR TUTORS
20%
Decrease In Time Used To Explain Math Concepts
Tutors' self reported survey data after one month of Co-Pilot usage.
RESEARCH
How did we navigate through ambiguity?
I began our research by reviewing 12 recordings of current tutoring sessions. Drawing insights and questions from this, I facilitated 2 focus group interviews with tutor leads and supervisors, mapping their pain points and hidden needs into a tutor journey map.
Goal
What does a normal tutoring session look like?
What are the relationship and interaction patterns between students and tutors?
Goal
What are some challenges tutors are currently facing and why?
Which part of the tutoring process needs the most support?
Goal
What does a normal tutoring session look like?
What are the relationship and interaction patterns between students and tutors?
Goal
What does a normal tutoring session look like?
What are the relationship and interaction patterns between students and tutors?
Goal
What are some challenges tutors are currently facing and why?
Which part of the tutoring process needs the most support?
Based on the research, I outlined the actions tutors take and the challenges they encounter before, during, and after a session on a tutor journey map to provide a holistic overview before focusing on a specific area.
Tutor Journey Map
Centering our product during tutoring sessions
To identify the stage of the tutoring journey with the greatest opportunity, I mapped the current support PLUS provides to tutors at each stage and found that the "during tutoring" stage has the least support.
PROBLEM STATEMENT
How might LLMs help PLUS tutors effectively and engagingly teach math concepts by providing in-session support that addresses their most critical needs?
IDEATION
200 ideas under 2 hours through careful prompting.
Our design manager challenged us to generate high-quality ideas within 2 days. Faced with detailed needs from our research, we experienced design fixation using conventional methods like Crazy 8s and Creative Matrix.
To break out of the fixation, I proposed that we take on the approach to leverage generative AI for ideation, meticulously refining prompts through simultaneous input-output evaluation.
Raw vs. Cooked, what's the difference?
Unpolished
Overlong
Not LLM-Powered
Vague
Irrelevant
Detailed
Actionable
Relevant
LLM-powered
Impactful
How did we cook up the input for better output?
My manager's take on our design process.
%20Tutor%20Co-pilot%20_%20Card%20586%20_%20T.jpg)
PRIORITIZATION
Picking the final design directions through data.
After sorting through 200 ideas using affinity diagramming and a gut check, we landed on 9 promising design directions. To refine these further, I created a survey for tutors to assess the relevance and helpfulness of the ideas and landed on the final 2 design directions.
Survey with 9 scenarios present in a Problem-Solution-Resolution format
Scatter plot with 9 ideas based on perceived relevance and helpfulness
Socratic Method Assistant (S6)
A chat-based tool that uses LLM to pose strategic questions that lead students to discover math concepts independently.
Step-By-Step Guide (S4)
An LLM tool that identifies a problem guide and generates a step-by-step guide with tips for tutors on how to present each step.
Landing on the final 2 design directions with high scores in both criteria
RAPID PROTOTYPING
Co-design GPT with stakeholders for ideal model output.
Now that we have two design directions, we face two challenges attempting to explore the desired model output.
CHALLENGE 01
We don't know what we don't know.
Gathering feedback for a futuristic experience that tutors have never encountered makes it likely to be unreliable to ask them to imagine and describe their desired model output.
CHALLENGE 02
High cost.
Building a functional prototype for users to envision and suggest improvements intensive in both resource and time. It may also diverge from what users actually want despite the effort.
Inspired by the recent surge of AI tools, the design team thought about a solution that can perfectly solve the above two challenges - co-designing the GPT in real time. I facilitated a total of 5 individual workshops with our end users, ranging from tutors, tutor leads, and tutor supervisors.
INSIGHTS
Initial shape of the ideal model output.
01
Fusing The Two Models
Switching between the two models can be time-consuming, given the limited time tutors have with each student. Combining the two models into one will likely to reduce unnecessary hustle.
02
Words Of Encouragement
In addition to the solution guide and leading questions for each step, including words of encouragement that tutors can directly use with students would be beneficial for extra motivation.
03
Table Format Output
Merging the two models with words of encouragement may make paragraph-based output overwhelming and less scannable. Presenting the information in a table format will enhance readability.
04
Emojis For Emotional Touch
Emojis are added after each word of encouragement because they can evoke positive emotions for tutors and students. They can also making messages more engaging and relatable.
ITERATIONS
Spearheading the model feedback cycle.
To ensure continuous model improvement, I facilitated communication between tutor supervisors and machine learning engineers by 1). outlining specific feedback gathering requirements for supervisors and 2). creating two rounds of iteration suggestion documents for engineers.
01
Crafted Feedback Requirement Guide For End Users
I prepared 8 pairs of input-output and asked the end users to comment with the following example guided questions "What’s working? 👍 What’s not working 👎 and how can we make it better?"
02
Translating Feedback Into Actionable Iteration Suggestions
I gathered feedback from multiple stakeholders, identified common themes, and transformed them into actionable iteration suggestions for machine learning engineers, following a 'good example' and 'bad example' format after each suggestion.
REFLECTION
My learnings designing AI products.
01
Designing AI products is an adventure.
I've discovered that creating AI solutions involves diverse modalities and skill sets compared to conventional product design. For instance, GPT co-design follows an iterative process of rapid prototyping, with real-time feedback and iteration vastly improving efficiency. Initially, this unfamiliar design approach seemed daunting, but exploring these new modalities has been incredibly rewarding. It's made me a more adaptive, well-rounded, and robust designer.
02
Speak up, sync up.
Due to the innovative nature of the design process for this particular product, I paid extra attention to communication with stakeholders. I sent frequent coordination messages to provide progress updates, set clear expectations about how they could contribute at each stage, and explained in detail the rationale behind each design decision. This proactive communication helped build trust and kept the project moving smoothly despite the ambiguity of the project space.
Next Project 🌱 👉
