Designing Virtual reality educational experiences

virtual reality
3D prototyping
healthcare
nuclear engineering
storyboarding
script designing
Educational experiences in VR
The XR Initiative at the University of Michigan has been funded to leverage emerging XR technologies to strengthen the quality of a Michigan education, cultivate an interdisciplinary scholarly community of practice at Michigan.
I was selected as an XR design fellow where I contributed to the vision of building training modules for the following courses:
  • Chemotherapy simulation for School of Nursing
    This training would help to train future medical students to handle the case of drug leakage into the tissues and cell structure
  • Nuclear Reactor experiments for School of Engineering
    A VR-based nuclear reactor learning module that offers training without the safety concerns of a live reactor
  • Physics XR Laboratory for School of Engineering
    A VR based spaceship mission simulation to help students apply physics concepts (momentum, gravity, and energy) in the experiments
My Role
Extended Reality Designer
Team
Toshi Gupta
Moeezo Saleem (XR Developer)
Shalaunda Reeves (Learning experience designer)
Duration
8 weeks (June 2020 - Aug 2020)
Tools
Paper pen sketch, Procreate, Figma,
Adobe After Effects
Core responsibilities
  • Design the scripts for all training modules by deep diving into the lecture and lab material
  • Sketch 2D storyboards of the script
  • Design 3D storyboards and design 360 degree views using motion design and video editing
  • Collaborate with VR developers to convey the designs to be developed
  • Devising a seamless pipeline of scripting, storyboarding, designing and developing VR experiences
UX pipeline for Vitual Reality
This whole process of collaborating with academia and building a product to be used by over ~200 students at the different schools of University of Michigan involved a lot of stakeholders. Right from providing the vision and steps arounf the hands on training the professors wanted students to perform to actually deplying those experiences in the XR studios, it was all a very iterative process.
Sketching the experience
After getting the script from the learning experience designer, I visualized and sketched the whole story. As a team, we went back and forth the script and filled gaps and brainstormed on new ideas. When I started prototyping for VR, I was thrown away by the lack of resources available. So I started collecting and trying out whatever material was available for rapid prototyping in VR.
Stage 1
Sketching 2D storyboards
After I got the script from the learning experience designer, I visualized and sketched the whole story. As a team, we went back and forth the script and filled gaps and brainstormed on new ideas.
Starting storyboards for chemotheraphy training being developed on Oculus Go
Stage 2
Designing 3D storyboards
Something had to be different and more about VR designs, right? When I communicated with the developers, I felt only 2D storyboards wouldn't do the job. As a designer I should be delivering things with a clearer and different perspective. I came across the blog bySaara Kamppari-Millerand Volodymyr Kurbatovfrom where I pulled different templates that bring more 360 degree & 3D perspective to the storyboards.
Templates used for calculating distances and comfortable viewing angles
My 3D sketches for understanding the object distances and room orientation
Stage 3
Designing for body movements with video editing
Combining role play and simulation to try out the actual experience is one of the great ways to start visualizing things. Right from performing actions using controller to walking or teleporting yourself in the VR environment, conveying immersive experience in a story board is a challenge.
Since I was in a remote setup, I had to design body storming results to convey it best to the developer and so, I tried my hands on in panoramic views and video editing. I put my storyboards in sequence. I used a panorama viewer to record the movement of my storyboard and then tried some effects using Adobe After Effects
Panoramic and 3D perspectives
Stage 4
Comparing with developed versions
and iterating
As a team, we decided to discuss and compare the design and final developed versions after every sprint. That provided me a check on my visualizing capabilities and what is not working out in a virtual enviroment
My storyboards being converted into actual experience in Oculus Quest for Nuclear Engineering class by Moeezo Saleem
Future work
The three modules are currently being developed and the designs are being worked upon. The plan is to launch these labs in the curriculum by the end of 2020.
Learnings
This has been a very exciting and challenging opportunity for me. I thoroughly enjoyed designing 3D perspectives and collecting resources around how to prototype in VR.
  • Impact in a remote setup:
    I am working fully remote for this position. Collaboration with faculty, developers and script writers has been challenging but following a definitive process has helped hugely.
  • Navigating in a niche field
    Sketching, prototyping and designing resources in the field of XR are not very straightforward. To find the best methods to convey my designs to developers and understanding constraints of 2D and 3D prototyping has been a huge learning. In this process, I read a log of blogs, company forums and talked to experienced designers in the industry to move in the right direction.
  • Documenting my findings
    This process helped me find a lot resources to look at and try out for prototyping for VR. I made sure todocument my explorations.
  • Planning for future
    This experience has definitely made me passionate about the future of design in AR/VR. I have also taken up teaching the course as a Student Instructor for the course Intro to AR/VR. My next step to my prototyping would be to using Unity and Unreal. I have done afew projectsin Unity and it is a great tool for designers who can code to communicate the designs with the developers
  • There is no limit to your imagination and its execution!
01
Team and Role
My role
Designer
Developer
Team
Toshi Gupta
Ruchita Lodha
Duration
8 Weeks
Mar'20-May'20
Tools
Unity
Vuforia
ARCore
Figma
02
Motivation
Dyslexia is a language based learning disorder which is found in 1 in 5 students and kids who have difficulty often avoid reading because it's stressful.
We wanted to use an immersive technology like AR/VR/MR for good together with digital game based learning to enhance the learning experience and help students retain things in a less stressful and playful way.
03
Problem Statement
To build an accessible learning experience for kids diagnosed with dyslexia between the age of 3-6 years. To help kids be less stressed mentally while learning to read and write alphabets.
04
Challenges in a classroom setup
There are specific set of challenges faced by dyslexic children in a classroom setup:
1. Manual Learning: Lack of multimedia based interaction to address learnability issues.
2.Individual learning approach: Teachers are less focused on students who have dyslexia in a classroom setup.
3. Less variation in learning: Lack of use of variety of senses such as seeing, hearing, talking, and touching.
05
Our Solution
To build an augmented reality game-based alphabet learning application for kids between age 3-6 years by augmenting the real world objects around them. The application would help students learn alphabets with phonetics and associated real world objects, identify and differentiate between alphabets and write them.
06
Why Augmented Reality?
We did a qualitative research on ways and methods adopted to help dyslexic kids learn faster like the Orton Gillingham method and the Wilson Method. We studied the online videos available  for teachers to pick out the key components that are very important while helping in dyslexia.
We realized thats supporting these methods with augmented reality for multi-sensory learning (Kinesthetics, Visual, Tactile, Phonetics) coupled with game based learning would help students get accurate and faster in alphabet identification
07
User Journey Mapping
Potential users: Kids who find it difficult to read, learn and identify alphabets.
The idea is to build an application that augments the alphabet cards/alphabet book with 3D holograms and support the phonetic learning through sound assistance. It will have 3 main features:
1. Learning about the shape and usage of alphabet
2. Identification of alphabets after learning through games
3. Learning to draw the alphabets through guidance
08
Lo-Fi Sketches
09
Introducing JoJo - the voice assistant
‍To help the students engage and navigate better with the learning and games, we introduced a cute animated hippo!
The main reasons behind having a voice assistant are:
1. Help students navigate between different parts of the application
2.Lower the burden of reading instructions for dyslexic kids
3.Give a comfortable learning experience through their "Friend- JoJo"
10
Technology decisions
1. The application is developed in Unity and tested on an android and iOS mobile device
2. To enable enhancements on a book/alphabet card, we decided to go for a marker-based approach in Vuforia

*This project was carried out during COVID-19 and therefore our plan of enabling the application on Hololens could not go through.
11
High Fidelity prototype and interactions
2D menu and instructions: The game starts with a 2D menu with an introduction by a friendly assistant JoJo. JoJo guides the user to the different sections of the application like Learn, Play and draw.
You can listen to JOJO's instructions in the final video attached.
Dragging/dropping and success/failure animations: To test students' capability to identify the learned alphabet, they are asked to drag and drop the associated 3D hologram into the ring options. Based on the answer the application reacts gently with animations of right and wrong answer.
Shooting with a ball: For letter identification, the application is supported with physics enabled bowling game. The students' are asked to shoot the associated hologram to the right letter glasses.
The right and wrong answer are distinguished with sounds.
Drawing: The students can reference the drawing technique animation and try drawing with a physical marker.
12
Final video
12
Reflection and learnings
Thanks for scrolling!