Designing virtual reality based educational experiences for students

VR Design/ Storyboarding / 3D prototyping/ Script writing
Educational experiences in VR
The XR Initiative at the University of Michigan has been funded to leverage emerging XR technologies to strengthen the quality of a Michigan education, cultivate an interdisciplinary scholarly community of practice at Michigan.
I was selected as an XR design fellow where I contributed to the vision of building training modules for the following courses:
My Role
Extended Reality Designer
Toshi Gupta
Moeezo Saleem (XR Developer)
Shalaunda Reeves (Learning experience designer)
8 weeks (June 2020 - Aug 2020)
Paper pen sketch, Procreate, Figma,
Adobe After Effects
Core responsibilities
  • Design the scripts for all training modules by deep diving into the lecture and lab material
  • Sketch 2D storyboards of the script
  • Design 3D storyboards and design 360 degree views using motion design and video editing
  • Collaborate with VR developers to convey the designs to be developed
  • Devising a seamless pipeline of scripting, storyboarding, designing and developing VR experiences
Product pipeline for Virtual Reality
This whole process of collaborating with academia and building a product to be used by over ~200 students at the different schools of University of Michigan involved a lot of stakeholders. The vision and course material came from the faculty which was researched and aligned thoroughly by the learning experience designer to write the script of the experience. The script was then communicated to the interaction designer (Me) where I worked to convert these ideas into storyboards, 3D prototypes and made important spatial design decisions. These designs were then communicated to the VR developers for production le vel applications.
Designing the experience
After getting the script from the learning experience designer, I visualized and sketched the whole story. As a team, we went back and forth the script and filled gaps and brainstormed on new ideas. When I started prototyping for VR, I was thrown away by the lack of resources available. So I started collecting and trying out whatever material was available for rapid prototyping in VR.
Stage 1 - Initial ideas
After I got the script from the learning experience designer, I visualized and sketched the whole story. As a designer who is taking the first take on visualizing this virtual reality experience, I spent some time to sketch my initial ideas to get to a point where I am convinced with the first round of user flow.
The basic ideation low-fi sketches for asking right questions
Stage 2 - Sketching 2D storyboards
As a team, we went back and forth the script, filled gaps in the flows and brainstormed on new ideas. As the ideas were more streamlined, I started working on sketches that would be used to communicate with the developer. I incorporated annotations and perspective from different angles which would help the developer save time on thinking of edge case scenarios.
Starting storyboards for chemotheraphy training being developed on Oculus Go
Stage 3 - Designing 3D storyboards
Something had to be different and more about VR designs, right? When I communicated with the developers, I felt only 2D storyboards wouldn't do the job. As a designer I should be delivering things with a clearer and different perspective. I came across the blog bySaara Kamppari-Millerand Volodymyr Kurbatovfrom where I pulled different templates that bring more 360 degree & 3D perspective to the storyboards.
Templates used for calculating distances and comfortable viewing angles
3D sketches for understanding the object distances and room orientation
Stage 4 - Designing for body movements with video editing
Combining role play and simulation to try out the actual experience is one of the great ways to start visualizing things. Right from performing actions using controller to walking or teleporting yourself in the VR environment, conveying immersive experience in a story board is a challenge.
Since I was in a remote setup, I had to design body storming results to convey it best to the developer and so, I tried my hands on in panoramic views and video editing. I put my storyboards in sequence. I used a panorama viewer to record the movement of my storyboard and then tried some effects using Adobe After Effects
Panoramic and 3D perspectives
Stage 5 - Comparing with developed versions and iterating
As a team, we decided to discuss and compare the design and final developed versions after every sprint. That provided me a check on my visualizing capabilities and what is not working out in a virtual enviroment
Storyboards being converted into actual experience in Oculus Go for Med class by Moeezo Saleem
My storyboards being converted into actual experience in Oculus Quest for Nuclear Engineering class by Moeezo Saleem
Future work
The three modules are currently being developed and the designs are being worked upon. The plan is to launch these labs in the curriculum by the end of 2020. A few of these projects are funded by Oculus at Facebook Reality Labs.
This has been a very exciting and challenging opportunity for me. I thoroughly enjoyed designing 3D perspectives and collecting resources around how to prototype in VR.
Team and Role
My role
Toshi Gupta
Ruchita Lodha
8 Weeks
Dyslexia is a language based learning disorder which is found in 1 in 5 students and kids who have difficulty often avoid reading because it's stressful.
We wanted to use an immersive technology like AR/VR/MR for good together with digital game based learning to enhance the learning experience and help students retain things in a less stressful and playful way.
Problem Statement
To build an accessible learning experience for kids diagnosed with dyslexia between the age of 3-6 years. To help kids be less stressed mentally while learning to read and write alphabets.
Challenges in a classroom setup
There are specific set of challenges faced by dyslexic children in a classroom setup:
1. Manual Learning: Lack of multimedia based interaction to address learnability issues.
2.Individual learning approach: Teachers are less focused on students who have dyslexia in a classroom setup.
3. Less variation in learning: Lack of use of variety of senses such as seeing, hearing, talking, and touching.
Our Solution
To build an augmented reality game-based alphabet learning application for kids between age 3-6 years by augmenting the real world objects around them. The application would help students learn alphabets with phonetics and associated real world objects, identify and differentiate between alphabets and write them.
Why Augmented Reality?
We did a qualitative research on ways and methods adopted to help dyslexic kids learn faster like the Orton Gillingham method and the Wilson Method. We studied the online videos available  for teachers to pick out the key components that are very important while helping in dyslexia.
We realized thats supporting these methods with augmented reality for multi-sensory learning (Kinesthetics, Visual, Tactile, Phonetics) coupled with game based learning would help students get accurate and faster in alphabet identification
User Journey Mapping
Potential users: Kids who find it difficult to read, learn and identify alphabets.
The idea is to build an application that augments the alphabet cards/alphabet book with 3D holograms and support the phonetic learning through sound assistance. It will have 3 main features:
1. Learning about the shape and usage of alphabet
2. Identification of alphabets after learning through games
3. Learning to draw the alphabets through guidance
Lo-Fi Sketches
Introducing JoJo - the voice assistant
‍To help the students engage and navigate better with the learning and games, we introduced a cute animated hippo!
The main reasons behind having a voice assistant are:
1. Help students navigate between different parts of the application
2.Lower the burden of reading instructions for dyslexic kids
3.Give a comfortable learning experience through their "Friend- JoJo"
Technology decisions
1. The application is developed in Unity and tested on an android and iOS mobile device
2. To enable enhancements on a book/alphabet card, we decided to go for a marker-based approach in Vuforia

*This project was carried out during COVID-19 and therefore our plan of enabling the application on Hololens could not go through.
High Fidelity prototype and interactions
2D menu and instructions: The game starts with a 2D menu with an introduction by a friendly assistant JoJo. JoJo guides the user to the different sections of the application like Learn, Play and draw.
You can listen to JOJO's instructions in the final video attached.
Dragging/dropping and success/failure animations: To test students' capability to identify the learned alphabet, they are asked to drag and drop the associated 3D hologram into the ring options. Based on the answer the application reacts gently with animations of right and wrong answer.
Shooting with a ball: For letter identification, the application is supported with physics enabled bowling game. The students' are asked to shoot the associated hologram to the right letter glasses.
The right and wrong answer are distinguished with sounds.
Drawing: The students can reference the drawing technique animation and try drawing with a physical marker.
Final video
Reflection and learnings