Eye Tracking Study
Background
The main purpose of this project was to learn about how humans experience design by tracking it through the use of eye tracking and other biosensors to measure their physiological responses. Along with my three colleagues we designed 3 separate studies where eye tracking and biosensors came into play in proving our hypotheses. Then we gathered the data and analyzed whether or not our questions were answered.
Project Set Up
In my group of 4, over a month period we worked together to create and run usability tests on 5 separate subjects for our small project. We created three separate tests, one on Elite Daily, a social news website, a comparison between product pages on Sephora and Ulta Beauty and finally a test of emotional response to a Reeses advertisement video. Afterward, we analyzed the data to locate significant test results that were of interest.
Before starting our research project we utilized this book, Eye Tracking the User Experience to aid us with coming up with potential question ideas and how to analyze the data we found through testing. This book also taught us about the different technologies we were going to use for the testing.
Tools We Used
iMotions software is a hardware agnostic platform that synchronizes and integrates biofeedback sensors into tracking the human experience. The software collects data and visualizes it to be analyzed.
We used this software to follow the Eye tracking, GSR and Facial Expression Analysis of all of our subjects.
This is the eye tracker that we used. The eye tracker is a sensor that can detect a persons presence and follow what they are looking at in real time.
The technology converts the eye movements into data that contains information like where the pupil is and gaze points.
This is a Galvanic Skin Response(GSR) sensor which records the sweat secretion from a persons hands. Sweating can be triggered by emotional responses to certain stimuli. The higher the level of perspiration on the skin, the greater conductance of electrical currents which is what the GSR records.
Research
Research Questions
Our first step in research as a team was to come up with a list of questions that we could ask for each of the different studies we were conducting. We did so through the use of examples from the book we read Eye Tracking the User Experience and also analyzing each of the websites we looked at.
Sephora vs. Ulta
How do the differences between the two beauty sites’ presentations inform user experience in terms of the findability and recognizability metrics?
How do the differences between the two beauty sites' presentation of product contents inform user experience in terms of findability metrics as well as any dwell time on possible distracting AOIs (the highlights and details section)?
How do the differences between the two beauty sights presentations inform user experience in terms of the findability and recognizability metrics, alongside fixations on AOIs that could interfere with completion of the task (All Reviews Section)?
How do the differences between the two beauty sights presentations inform user experience in terms of the findability and recognizability metrics?
How do the differences between the two beauty sights presentations inform user experience in terms of the recognizability metrics as well as overall site presentation/organization and the affect presentation on valence?
Elite Daily
Does the current presentation of Elite Daily’s Website uphold user experience when searching for current trending topics in terms of fixations on distracting AOIs as well as target findability?
Does the current presentation of Elite Daily’s Website uphold user experience when locating the astrology section on the homepage in terms of target recognizability and distracting AOIs?
Does the current presentation of Elite Daily’s Website uphold user experience when navigating through the menu section on the homepage to find the food button in terms of target recognizability and findability?
Does this section appeal to user experience in terms of emotional valence?
Reeses Advertisement
Tell us what you thought about the commercial?
What is your favorite part?
How did you feel while watching it?
Script writing: Creating Tasks
For the second part of this project we worked on developing a script with tasks for each of the different tests we were going to run. Together, our team went through each of the websites and came up with real tasks that someone who was on that page would do regardless of being in a study or not. We did not need tasks for the advertisement as we were just tracking emotional responses.
Creating tasks is not as easy as it seems. When our team was working on writing these tasks we wanted to make sure that we did not use any language that could skew the outcome of the test. An example of this was when we switched the word “ingredients” to “components” so that we did not let the user know exactly what they were looking for.
For each of our different website studies we employed a different type of think aloud. For Elite Daily we asked the participants to talk about what they were thinking while they completed a task on the site. The tasks for Elite Daily were mainly focused on how the site was structured. We wanted those first actions that they were planning on taking while they were acting on them.
When we did the comparison of sites for Ulta Beauty and Sephora we used retroactive think aloud where we asked questions after the tasks were preformed so that the user could first completely focus on the task and then start to explain what they were looking for and why they took those actions.
The Tasks
Elite Daily
Find what is trending currently and read off a title from the trending list. Let me know what you think of this section?
Locate the astrology section on the homepage and tell me when you find it. Tell me what you think of the section.
Locate the fall section on the homepage and tell me when you find it. Now tell me what you think of the section.
Go to the menu section on the homepage, look through it and find the food button. Tell me what you think about the menu?
Sephora vs Ulta Beauty
You have a concern about the product you want to settle before you consider ordering the face mask. Locate where you would ask a question about the product and tell us when you’re done.
Now you want to know what the contents are in the product. Look around the page and find the contents list. Feel free to click where you see fit and tell us when you’re done.
You are interested in seeing pictures from customers who have used and reviewed the product. Locate the product reviews with pictures and tell us you’re done.
You are interested in purchasing Glam Glow for your skin; however, you don’t have the $60 upfront to purchase it. Locate where you would find financing options for the product; tell us when you’re done.
Scroll through the page one last time and provide a score out of ten based on how cluttered you think the page is, one being not cluttered at all and ten being horribly cluttered.
Running the Tests
After completing our scripts we began to bring in our test subjects in and outside of class time. We would go in pairs to run the tests along with the test subject. One of our team members would execute the test this means telling the test subject what they were going to be doing, sitting beside the subject and read them the script that we had written. The other team member would sit and observe the test session and take any interesting notes down based on what the user was doing. The test subjects were not completely random each person from our group picked one to two people that they knew to preform the test.
Analysis
After the tests were all complete we received all the videos that were taken with the iMotion software. The videos showed the gazes of each of the participants for every task that they preformed. As a group, we looked at the data and carefully watched the videos to see where the users succeeded and where they failed. We used areas of interest(AOI) that we discussed with our professor to get metrics for if our test subjects found those areas or not. Areas of Interest are the areas on a webpage that we as a team decided were of statistical significance. The iMotions software allows us to circle the AOI on the webpage and then when the user looks at that AOI it tracks it, it tells us how many times the test subject looks at the AOI, total time they dwelled on it and how long they first looked at the AOI.
Findings
Elite Daily
Finding 1: Poor Scannability of information
5/5 Participants Reported a Struggle to Read a Title from the Trending Section
When the test subjects began to look at the trending section the movement of the titles made it difficult for respondents to read off a title. The test subjects all showed low fixation duration, they would lack focus on multiple titles and often not know where to focus their eyes to read off.
“I think it is way too much, especially moving in opposite directions"
- Brianna
It was easy to find but I also don’t like that it stacks up and that it’s moving because when I was trying to read, they were already gone.”
- Alyssa
“This is a lot harder to read than Twitter’s Trending List”
- Jackson
The automatic menu is another potential area of poor scannability, but there wasn't enough data beyond speculation to support this as comprehensive evidence.
We had a user struggle to locate the AOI due to the drop down menu being automatic, we came to this conclusion as a general opinion of the group when looking over the data that we collected and interpreting the users behavior with one user even saying:
“ Ooo this scrolls over very fast, oh man, I don’t really love that it opens and closes as you put the mouse over it” - Alyssa
Finding 2: Elements of interest were in Unexpected Locations
The size of the astrology title caused an initial target comprehension error for 3/5 of the users during their search.
A lot of users could locate the target area but, they could not fully comprehend that the astrology section was the actual astrology section, they would find it and then scroll past or ignore it completely at first. This is an example of poor target recognizability, the users could not comprehend the astrology section at first.
“A little blank, it’s very tiny. If they are going to make this a part of their website they should flesh it out a little more.”
-Jackson
“I think that’s all of it.”
- Brianna
“This looks like the astrology section, but I don’t know, I am gonna keep scrolling”
- Alyssa
3/5 participants disagreed with the location of the 'food' button in the menu.
it took our subjects a considerable amount of time to have their eyes fall on the target that we picked out and showed an immediate dislike to its placement.
“I would not expect to find ‘Food’ under the news section. I would expect to find it under ‘Experiences’ or ‘Entertainment’ sections before ‘News’”
- Brianna
“Everything is going really fast”
- Alyssa, skipping over the designated section twice before finding it
“Now I feel stupid”
- Jackson, after spending time looking through every other section before locating food under the news section
Sephora vs Ulta Beauty
Finding 1: Target Findability was superior on Ulta
4/5 users scrolled past the area of interest, failing to locate the "Ask a question" button at the top of the page on Ulta.
On Sephora, 5/5 users scrolled past the area of interest, failing to locate the "Ask a question" button at the top of the page.
Molly on Ulta was the only subject to locate the ask a question AOI, taking 8 seconds to do so. This conveys that there is possible comprehension problem with the AOI.
4/4 participants easily and successfully located alternative financing options for Ulta, while only 3/4 located them on Sephora.
One participant misunderstood the task across both sites.
Another participant located alternate shipping options instead of alternate payment options.
Our 5th participant was tripped up by our tasks wording and located alternative products to Glam Glow instead of alternative financing options for the product so they have been excluded from the study. This was due to a misunderstanding of what the task was. The target was not well understood by the users, they often fixated on the subject but then move on to find an alternate option.
Finding 2: Presentation of Information was better on Sephora
All 5 participants struggled to locate reviews with pictures on Ulta and were able to easily locate them on Sephora
This task was difficult in nature because the sites had different locations for reviews with pictures on the site so we measured the amount of time it took each participant to locate the pictures on Ulta. This was a big Target Findability error and Recognizability error, the fastest time on the Ulta website was around 48 seconds to find the target. One user Brianna completely failed the task having to be stopped at 3 minutes because she just started guessing. In the video you see here you can see how this user, Molly skips over the images section in the dropdown menu completely.
Comparatively to Sephora where it took around 10 seconds for users to locate the reviews with photos and all 5 located the correct AOI for Sephora.
Reeses Take 5 Super Bowl Commercial
Finding 1: The first joke in the video caused the strongest reaction, but caused the users to not have any reaction for the others.
The users show a strong response to the first joke in the video but their responses emotionally cease after the first joke.
you can watch the commercial here
Only 1/5 people understood the final pun, which was the real punchline. Meaning a lack of comprehension of the joke, this could be because of not hearing the phrase before and having to understand just based on presentation of the joke.
"It was funny how they were recreating common phrases in real life. It was funny how the last one wasn't explicitly said but you can guess what it meant" -Thomas
One participant didn't understand any of the consecutive puns at all.
"I didn't get the naming different things" -Alyssa
Overall though majority of the responses were positive to the commercial which was the intent. The purple bar on the graph represents the amount of positive frames that the user had and is separated by user. This can also be seen by our user statements after watching the video we asked them how they felt.
" I felt really happy, I thought it was funny"
- Molly
"I felt awfully silly watching this commercial. I thought it was really silly. "
- Alyssa
"I felt good - it was really funny"
-Thomas
Outcome
Throughout this assignment the most difficult part was analyzing the data after we had conducted the Sephora and Ulta study. This is mainly because the two websites were fairly similar in placement of Areas of Interest and users had almost identical responses to both of the sites when it came to locating AOIs from our tasks so we really as a team had to watch through the gaze videos multiple times to fully understand what our users were doing. The difficulty was we would find an issue in one of the users responses but often when we looked if there was ever a repeat of the same or similar behavior for another user there would not be.
For this project I learned a lot more about what eye tracking can be used for and how it ultimately helps when it comes to certain sites and certain AOIs that we are looking for. I also had the privilege of helping run a test for the study, it was educational to sit and guide the user through the tasks that I wanted them to preform and also get a bit of a closer look at iMotion and also the Eye tracker and the GSR machines. I even got to try them on myself in a separate study where I got to be the test subject which was a lot of fun to actually try out the technology myself because we did not get to try it in class.I see why Biometrics and Biosensors are very useful for studying human experience they get to show what the person is actually doing instead of what they may be saying, it helps you get a look at how the user is really thinking just by the movements of their eyes. You get to see how their eyes are caught by certain places on the page first or when the eyes don’t know where to look they move around more you get to see how the brain is processing the page while also trying to complete the task.