Human Machine Interaction (HMI) with Augmented Reality in a Family Vehicle 

Asset 1.png


What is the future of integrated augmented reality systems in vehicles? We were sponsored by the Human Machine Interaction (HMI) Lab at the Georgia Institute of Technology to research how augmented reality integrated windows could benefit people. We aimed to enhance family trip experience through the use of a family car that utilizes augmented reality windshield/windows. Our focus was on encouragement of engagement and the mitigation of pain points between passengers and drivers through an infotainment system in a family trip context. 


Beatrice Adebisi, Samira Bandaru, Iris Colendrino, Abigail Maeder 


Human Machine Interaction (HMI) Lab at the Georgia Institute of Technology

Personal Research Role

Led Design Research, Interviews, Task Analysis, Observational Analysis, Usability Testing, Journey Mapping, Insight Generation, UX/UI Design


9 Weeks    Sept. 2018 - Nov. 2018

*NDAs and confidentiality agreements were signed during the course of this project. Not all project and personal details are shown.

Project Overview
  • Project statement:

    • We aim to enhance family interactions through the use of a family car that utilizes augmented reality windshield/windows. Our focus is on encouragement of engagement between passengers and drivers through an infotainment system in a family trip context. ​

  • The primary objectives of the project included:

    • Discovering and understanding family dynamics.

    • Discovering and understanding passenger interactions inside an automobile.

    • Create a testable and functional human-centered design that meets the needs of families.

    • Gain recognition for the HMI lab to attract future partnerships and sponsorships.

  • Research objectives included:

    • Understanding the environment, context of usage, and how users can benefit.

    • Empathizing with end-users and understanding their motivations and values.

    • Uncovering insights from observations, interviews, qualitative and quantitative analysis.

Methodologies Used
  • Observational Analysis

  • Stakeholder Analysis

  • Precedence and Standards Research

  • Task Analysis

  • User Interviews

  • Need finding & Need Hierarchy

  • Journey Mapping

  • Usability Testing & Analysis

  • Eye-Tracking Evaluation

  • User Identification

  • Affinity Mapping

  • Prototyping

Initial Research Process
  • We conducted stakeholder interviews to understand the goals of the project better. 

  • We reached out to and conducted expert interviews with members of prominent automotive manufacturers.

  • Several stakeholders, such as automotive companies, were identified and mapped to visualize priorities.

  • We distributed several go pro cameras to 6 different families that agreed to filming their family road trip.

  • Observational analysis based off of the videos and going on our own road trips with our respective families led to interviews.

  • Screening users resulted in 11 families/individuals that we conducted in-person interviews. Interviews were transcribed, recorded, and coded.

  • Journey mapping, task analysis, and user identification based on our initial research, observations, and interviews.

  • We began affinity mapping to see overall trends, needs, and desires of our users.

  • We researched existing precedents of "smart cars" and looking into governing standards and regulations that could potentially influence the design of the augmented reality system.

  • We created affinity maps to visualize potential trends and over-arching experiences shared by our users. 

  • Emotions and connections developed between family members are influenced by activities unique to the environment.

  • Family hierarchy, or roles, determine the individual’s perception of what a “road trip” is.

  • People’s most impressionable memories are created in the formative years (4-16).

Design Criteria
  • The system should be adaptable to situations with different modes that the user can switch between.

  • The system should adapt to and learn user tendencies to ease the amount of input the user needs.

  • All members of the family in the car should be able to connect to one another. The system should enable or encourage users to make a connection with each other. Whether it is by connected screens, conversations, or something else.

  • All users should be able to interact with the system. The system should have suitable options for people of all ages and the primary passenger should have access to all screens.

  • The car should have some level of automation (Level 3 Automation recommended) to allow driver to be involved by not inhibit the safety of the vehicle.

  • Navigational systems should be integrated and accessible.

  • Platforms of activities that involve both individual and multiple users to interact should be included. 

Prototyping, Testing, and Evaluation
  • We created and designed multiple testable interfaces.

  • We utilized a transparent touch OLED screen and car simulation buck to setup an immersive test.

  • Through developing screeners and reaching out to communities we found several families to test our designs.

  • We developed evaluation criteria based upon our design criteria.

  • Through use of Tobii Eye-Tracking equipment we could evaluate our primary user's focus during the experiment.

  • We conducted several exit interviews, self-evaluations, and system usability scales for the users we tested with.

  • We developed multiple distinct user interfaces with different UI elements and flows.

  • Our systems were successful in many of our design criteria but failed to clearly accomplish our primary goal of increasing engagement between all members of the family in the context of a family vehicle.

  • Our system proved to be successful in providing a personal space for our users and segmenting the individuals in the family.

  • The main advantage was keeping children occupied and subdued during times they were likely to be more rowdy/cause problems.

  • Our study was limited in both time and resources. The OLED screen that we used was costly and could only supply one or two of our users with a fully functional screen during testing at a time. This makes it difficult to truly understand the impact a full system would have. 

  • Testing and observing the system over the timeline of a road trip was also not possible. Most of our participants could only be testing for 1-2 hours with us.

  • We compiled our detailed findings into a slide deck and report that we presented to our sponsor, a board of researchers, and other stakeholders.

  • All research and development was handed over to the Human Machine Interaction Lab and research will be continued.

  • What I Learned:

    • Individuals have unique perspectives but people share similar views/interpretations of situations depending on their developmental stage.

    • People will take on the roles of their perceived label. Examples being of the shift in role when sitting in the driver's seat versus the passengers. Responsibilities assigned depend on the expectations of the user/family.

    • Qualitative data can uncover higher level personal needs that we can cater to in our designs.

    • Positive family interactions include much more than just our interface but we can use tools to encourage positive family interactions.

Understanding Objectives & Defining Goals

The design need for this project is to enhance the family trip experience through the use of a family car that utilizes augmented reality integrated into the car windshields and windows. The focus of this project is on the encouragement of engagement and the mitigation of pain points between passengers and the driver through an infotainment system in a family trip context. We focused on families with children between the ages of 4-16, as these are a child’s most formative years. These interactions were measured quantitatively through eye tracking technology and qualitatively through self-evaluation scales and rubrics, with criteria guided by experts in the field of psychology and human factors.

Project Research Questions:

  • What constitutes a family interaction/engagement and how does this change depending on the context/environment?

  • Do family hierarchies remain the same in the context of a vehicle?

  • How would an interactive augmented reality system operate through windows and windshields?

  • How could an infotainment system encourage interaction between passengers while mitigating pain points?

Detailed Process
Family Interactions and Needfinding

The focus of this project is to enhance the user experience of families during their road trips with our system. We need to understand how family dynamics influence and dictate the process of a road trip and how we could design to enhance this experience. 

Stakeholder Interviews and Identification

We spoke directly with the director of the Human Machine Interaction Lab to pinpoint business goals as well as identify stakeholders that would influence our project direction. Business goals included gaining attention for the HMI Lab through the project and seeking out potential sponsorships and partnerships with other automotive companies. Through plotting the potential stakeholders we began to see how they could impact our design. 

Asset 2.png

Stakeholder Matrix

Observations and Analysis

Observing a family road trip is a difficult challenge. Introducing an observer to such a small and intimate space would change behavior of the family and the outcome of the study. We were fortunate enough to have a few of our team members go on road trips with their own family and supply observational videos, pictures, and notes. We also found several volunteers to take Go Pro Cameras with them on their own family road trips. This camera placement was initially disturbing for our participants but footage showed the camera became a non-factor about 30 minutes into the drive and users were acting as if it was not present.The observations yielded important information about the hierarchy of the family and interactions that take place throughout the duration of a road trip.

Asset 3.png

User Interviews

Interviews for this project were very sensitive as we were asking people to tell us about their families, siblings, children, and some of their most precious memories. We found 11 different families/individuals to talk to about what made family road trips special to them. We focused on the experiences that they have today with their families, how those experiences compare to when they were kids, and what they hope their children remember about the trips they take today. 

Some sample questions from the scripts for our users included:

  • How would you describe your family dynamic?

  • How does your family dynamic change during road trips?

  • What type of traveler are you? Sight-seer? Destination oriented?

  • Who drives during the trip? Do you alternate drivers?

  • As your children have grown, how has their attitude towards road trips changed?

  • Do you enjoy road trips? Are they more stressful than fun?

  • What activities do you do as a family to feel connected?

  • Can you share a time you enjoyed yourself on a family road trip?

  • Can you share a time you had a negative experience on a family road trip?

We took the time to ask our users about their existing family dynamics, what a positive family experience means to them, and how they could envision a better experience in the context of a road trip. We introduced many of them to existing technology in the market today that could be similar to what our design could entail. These interviews led to in-depth discussions that we video recorded, transcribed and coded.  

Asset 2 (2).png


Task Analysis

We gathered our observations, first hand experiences, and our user interviews to construct a task analysis. 

Asset 4.png

Journey Mapping

After looking through our observation videos and interviews we created a journey map that captured the most shared experiences between our participants. Our journey map does not focus on a single individual, but rather on the entire family as a whole. We are looking specifically at the overall happiness, sadness, interaction and mood of the family throughout all these events that occur in the process of a road trip.

Asset 6.png

Target Demographic and Personas

By creating a visualization of the specific family members we identified our specific target demographic. We plotted the main phases of life and assigned general characteristics to each. Our focus on the family helped us clearly identify the age ranges, and characteristics. Persona cards were then created for each member of the 5 member family that hit each age range.  

Asset 8.png

Affinity Mapping

Asset 9.png

We combed through all of our gathered data at this point in our process and mapped out information, quotes, and observations we have seen with sticky notes. We identified 5 major areas of focus in our data.


Family Values:​

The core of our project is based upon what the family values. Which interactions are the "best" vs "worst"? Do all families have similar values? People value togetherness, but also want to have their own space. Some of the strongest memories we heard about from our interviews involved families following traditions, exploring new places, and conflict between each other. 

Anticipated Pain Points:

Many users went into detailed accounts of how they anticipate pain-points on their road trips and how they counteract these pain points. Some users pack extra toys for their children or food for their journey. Some bring headphones to drown everyone else out. Some plan the entire course of travel with pre-planned rest stops and dining options. It is all about overcoming these expected obstacles. How can we help users to avoid or overcome some of these pain-points?


Each family has their own unique hierarchy among its members. This hierarchy governs the entire system during a road trip. The passenger will sometimes have the greatest power as they can enforce their will on anyone in the car. Many users define the driver as in control. The existing family hierarchy outside the car does not always remain the same inside the car. and individual's perception of the road trip is heavily dependent upon their role in the system.

Family Activities:

This was the most positive experience that was reported by our users. Family activities and engagement created some of the most detailed anecdotes. While most of the experiences shared were positive, negative ones were also shared in great detail. These moments of interaction play a large role in defining the road trip experience whether they are as simple as the alphabet game or as complicated as card games.


Existing precedence and out stakeholders focused heavily on the prospect of how the introduction of autonomous vehicles can provide an opportunity to enrich the family road trip experience. By freeing up the driver the dynamic could completely shift in the context of a vehicle. Our users showed interest in autonomous vehicles but concern in the technology. People were often willing to try it themselves but not with their family. Trust and a limited level of autonomy would likely be a solution to integrate the feature into our project.

Insights and Design Criteria


  • Emotions and connections developed between family members are influenced by activities unique to the environment.

  • Family hierarchy, or roles, determine the individual’s perception of what a “road trip” is.

  • People’s most impressionable memories are created in the formative years (4-16).

Asset 12.png
Asset 13.png
Asset 14.png

Design Criteria

The opportunities that we discovered through our insights led directly to our design criteria. To accomplish our goal of encouraging family interactions we needed to include features that would enable the following criteria.

Asset 15.png

Evaluation Criteria


Recognizing that this car will be owned by an one family for around 7-8 years, it is a machine that needs to grow with the family. By introducing different modes for different ages and needs, families can customize the experience and change it as they grow. This design criteria was separated into two components: family and individual. The effect of these modes on each members of these categories was measured through self-evaluation scales and the duration of conversation rubric.

Learning AI/System:

Mitigating pain-points is a primary concern for our decision makers. The system should learn from previous interactions and learn from the patterns that families have. The system should propose different modes or activities to assist in mitigating expected pain points. Due to the limitations of this project and technology, this attribute was unable to be implemented, evaluated, and tested.


This criteria split up into three categories: screen sharing, memories and conversation. Evaluation criteria for screen sharing includes the amount of AR screen time versus time spent interacting with family members. Memories were measured with post-test self evaluations scales. Conversation was measured by a rubric developed based on the number of conversations with parents, siblings, eye contact and instances of mediation for each iteration of the testing simulation.

Complete Interactivity:

This means that every member of the vehicle should have the ability to access and interact with the system that enable or creates connection. Front AR access and back AR access was quantified by the number of interactions with the system and measured using the self-evaluation scales, Tobii Eye Tracking device and qualitative observations.


This refers to integrating some level of an autonomous system into the family vehicle in order to alleviate the responsibilities of some or all roles in the car. In the same way as the Safety/Trust design specification, this was an element that was not part of what we were testing, but still a crucial feature of our final concept.


Not only must we provide an opportunity for members of the vehicle to experience different roles, but how we establish that system must provide a safe means to do so and instill trust in the user. While this was one of the main criteria in the beginning stages of this project, we realized that regardless of what the final prototype was, it had to follow the rules and regulations of SAE and NHTSA and was therefore more of a mechanical specification than a design criteria.

Regulatory Criteria

  • The amount of light that passes through a car window has to be 70% or greater. [NHTSA]

  • Drivers are required to have a minimal amount of concentration and attention while in control of a conditionally automated car. (Eyes should not be off of what’s in front of them) [NHTSA]

  • At least three measurements be taken at different locations on the vehicle to determine the relative virtual image distance before being projected into the car [SAE]

  • A HUD (Heads Up Display) measurement system must perform luminance measurements on checkerboard images with alternating patterns to determine virtual image contrast for white and black projections in ambient light. [SAE]

UX Design and Prototyping

Design for an Augmented Reality (AR) format introduced a challenge none of us had previous experience with. The total size of the active area was an abnormal shape and required versatility to be put on windows of different sizes. Design considerations revolved around accessibility with anthropometric data (vision, reach, posture) and versatility of the interaction points of the UI.

Initial Wireframes and Design Considerations

Initial design consisted of defining features and navigation for the system. Iterations included various graphic options from the inclusion of highly visible colors to monotone, collapsible menus, gestural navigation, and remote control from a separate screen. The UI designs all include a black screen background. This is due to the hardware we used for testing. The transparent OLED touchscreen converts black to fully transparent and white to fully opaque. This makes use of color difficult to translate well, and became a design consideration quickly. Visually, we learned from displaying the graphics on the transparent OLED screen that the background of the screen should be black to provide transparency and text/icons should be white to reduce distractions.

Asset 19.png
Asset 20.png

Initial UI Designs

Testing Setup

Our testing setup was the AT&T car buck in the Human Machine Interaction (HMI) Lab, paired with a 55" transparent OLED touch screen as one of the windows for the backseat passenger. The car assumes level three autonomy, meaning that while the car is fully autonomous, the driver can take control at any time. An interactive interface combining learning and the outside environment was displayed on the OLED screen to keep children engaged and mitigate pain points during long family road trips. 

Asset 16.png

Testing Process

In testing, we analyzed the effect of our system on family interaction. To do this we simulated a control scenario and three different variable systems — one without any stimulation, one with the interface introduced on the OLED screen and one with a main screen on the heads up display where each user interacted with a secondary screen that was linked to the main screen. These interactions were measured with self-evaluation scales and Tobii Eye Tracking glasses that allowed us to compare gaze data from the OLED screen and the user’s family members.

Our testing protocol for the first prototype was designed to have the first 15 minutes as the control where the family would act as they would on a regular road trip without electronic devices. The next 15 minutes introduced the interface in which only one rear passenger had access to the interface. We measured family interaction by comparing the number of conversations and eye contact with the parents/siblings before and after the interface was introduced. The interface user wore Tobii eye tracking glasses in which we were able to analyze where and how long the user’s focus was. Self-evaluation questions were asked at the end of the simulation to provide user feedback.

Testing Prototype

The functional prototypes were made with ProtoPie and assets and graphics were made in Adobe Illustrator. The prototype shown is one of many that we had the opportunity of designing and testing.

Results and Evaluation

We learned from our first prototype testing that family interactions decreased due to more individual activities performed by the interface user. Family 1 consisted of a mother, a father, a 7 year old girl, and a 12 year old boy who interacted with the interface. During the control session, the family would hold conversations with each other, but the two children eventually started bothering each other and the mother would have to mediate. When the interface was introduced, the boy quickly became occupied with using the interface while the others would glance at the interface and make small comments, but had reduced conversations with each other.

For our second iteration, we added a “Family Mode” which is to be implemented at Level 3 autonomy to increase interactions through communal activities. Testing for the second prototype had an added 15 minutes for the family to play a shared activity together. Family 2 had a mother and two girls ages 7 and 8. This second round of testing had similar results to Family 1 where both families decreased in interactions with each other when the interface was introduced. However, when the family game was introduced, the number of
interactions were similar to the control. The Tobii eye tracker was used for Family 2 where heat maps were generated for the control, interface, and family game as well as gaze plots. The heat maps show where the user is looking at with the red color indicating the area with the most focus. We found that there was not as much attention on the road and the driver when using the interface compared to
the control. For the family game, the user’s attention was drawn to her family members’ game devices.

Eye Tracking Gaze/Heat Maps

We introduced our prototype to several families and analyzed the eye tracking information from all of the testing we conducted. We used the eye-tracking to evaluate the interactions between our family subjects, the amount of focuses specific attributes of the system held, and to gain a general understanding about the tendencies of our different users.


Test 2 Interface User - Focus Time Distribution

Asset 18.png

Result of the System

We tested the system with 5 different families and data was consistent throughout whenever the interface and family activities were introduced. We found the interaction between families significantly reduced once the introduction of our interface happened. This is likely due to the fact we had no way of interconnecting each individual interface during the testing period. When we switched to a family game that changed the  main interaction point to the main windshield, the number of family interactions increased. The number of interactions without the interface and with the family game were similar, but the quality of the family game interactions were higher quality with longer conversations, several moments of laughter, and consistent smiles between family members.

Asset 17.png

Interaction Breakdown

Conclusion and Moving Forward

The system we designed did not clearly accomplish our primary goal of increasing engagement between all members of the family in the context of a family vehicle. Our system was effective in mitigating pain points caused by children and providing individual space. We were able to identify and develop a family activity/game that we could project on the windshield that helped bridge the gap in the system. The failure of the system was directly due to the lack of shared interaction between the family members. Next iterations will incorporate a more detailed set of activities and further interaction between family members with additional screens for testing

Asset 23.png

Our team performed extensive need finding research to find a design opportunity within the realm of AR in family-oriented autonomous vehicles. The design opportunity is to encourage and enhance family interactions through the use of a family car that utilizes augmented reality integrated into car windshields and windows, where “interaction” refers to mitigation of pain points and the encouragement
of conversations. We interviewed groups that reflect our potential users, developed an ethnographic film, defined our design and mechanical specifications, developed low fidelity prototypes and evaluated them in order to narrow our focus, and after many iterations, developed our interface. These interfaces were tested with different families over a span of two weeks, which ultimately led us
to the conclusion that the presence of a main screen on the HUD that linked each passenger’s secondary screen was the best way to encourage interaction between family members using AR technology. We compiled our research and findings into a detailed report and slide deck to present to our client and other stakeholders. Research and development is being continued by the researchers at the Georgia Institute of Technology Human Machine Interaction Lab.

Other User Research Projects