Before the launch of the Avalon Mobile application, my team wanted to get some real user feedback. The goal was to discover any show stoppers and fix them before we launched the app. I was given a budget to recruit and test between 16-18 students enrolled in college level courses. Because my coworker had an established network of friends at Cal Poly in San Luis Obispo CA, it was determined that it was the best place to run user testing sessions. I was in charge of writing the script, the recruiting, moderating and delivering a finalized report.
ROLE: USER researcher
Recruitment
I ended up finding 16 students in total. Because our target user could be just about any student, the requirements for the study were very simple:
- Must be enrolled in college level courses
- Must be available one of the two days I was in town
- Should be an iOS user (this was because the iOS app was further along in development than Android)
- Preference goes to non UX or computer development majors
Script
I split the script into sections including:
- (2 min) Introduction: An intro to the study and thank participants.
- (5-10 min) Warm Up Questions: Briefly have participants walk me through current study tools they may use, both physical and online.
- (20-25 min) Tasks: The meat of the test. This is where participants are asked to walk through a full Avalon Mobile assignment including a sample reading and different question types about the reading.
- (5-10 min) Wrap Up Questions: Participants are asked for final thoughts and asked to rate the application's ease of use.
Key Findings
Overall students found the experience fairly easy to use. Most said that they liked having the textbook and questions in the same application and that they enjoyed the variety of question types.
When asked, "if you could change one thing what would it be and why?" students most commonly answered:
- Some kind of indicator that there was more content hidden below on the scrolling question
- A more intuitive way to use the matching question (or instructions on how to answer that question)
- Links back to passages in the book on the wrong answer screen for questions they got wrong
Students were asked to give a final rating of ease of use of the product. On a scale of 7 where 1 was very difficult and 7 was very easy students scored the experience as a 6.1.
Completed Assignment Screen:
Over half of the students I talked to mentioned that they would not read the "save time on your next assignment" because it felt "very general" and "like a lot of text".
Students liked seeing their most challenging topics. A few mentioned that they would want links to specific sections in the book to review.
Student's weren't sure how progress would change because, "wouldn't it always be 100%?".
Recommendations: Take out "save time on your next assignment". Take out Progress %. Add links to passages in the book to review.
App Overview/Tutorial:
13/16 participants said they would usually skip app tutorials.
Most participants tried to click on the answer to the sample question the app took them through rather than simply tapping to navigate to the next tutorial slide.
Recommendations: Re-work design to look like a tutorial so student's aren't as confused about where to click. Provide a way for students to get back to the tutorial in the case that they do choose to skip it.
Scrolling Answer Question:
There are four correct answers on this question, one of them requires the user to scroll down to find it.
The majority of students missed this question on the first try, some missed it on the second try as well.
“You can’t see the scrollbar on the side”.
"It’s like perfectly hidden. There’s no indicator that I have to scroll down".
Recommendation: Add a small down pointing arrow on questions where the answer choices may be out of vision.
Time for a Refresher Popup:
Students didn't always like that the application forced them to go back to reading depending on how they were doing.
"I'd try to power through it and just scroll down on the reading then go right back into the questions".
"I don't like it when apps think they're smarter than I am".
Recommendation: Make it optional for students to go back to reading, don't force them.
Matching Question:
This was probably the most severe usability issue that I uncovered.
Only one student was able to figure out the long press on the first try.
Students struggled to figure out how to use this question. Many tried tapping and dragging the images to match together.
Recommendation: Change the matching to a tap or drag instead of a long press.
Conclusion
I worked with the Product Manager to determine which of these issues we needed to fix first and what the correct fix would be. Things like the scolling question and matching question that were insurmountable to many students took priority over things like making the refresher message optional, which was more of an annoyance. Images from the final design can be seen in my project titled Smartbook.