Designing virtual reality based educational experiences for students
VR Design/ Storyboarding / 3D prototyping/ Script writing
HEALTHCARE GENOMICS
UX DESIGN
DASHBOARD DESIGN
UX INTERVIEWS
Educational experiences in VR
The XR Initiative at the University of Michigan has been funded to leverage emerging XR technologies to strengthen the quality of a Michigan education, cultivate an interdisciplinary scholarly community of practice at Michigan. I was selected as an XR design fellow where I contributed to the vision of building training modules for the following courses:
Chemotherapy simulation for School of Nursing
This training would help to train future medical students to handle the case of drug leakage into the tissues and cell structure
Nuclear Reactor experiments for School of Engineering
A VR-based nuclear reactor learning module that offers training without the safety concerns of a live reactor
Physics XR Laboratory for School of Engineering
A VR based spaceship mission simulation to help students apply physics concepts (momentum, gravity, and energy) in the experiments
Paper pen sketch, Procreate, Figma, Adobe After Effects
Core responsibilities
Design the scripts for all training modules by deep diving into the lecture and lab material
Sketch 2D storyboards of the script
Design 3D storyboards and design 360 degree views using motion design and video editing
Collaborate with VR developers to convey the designs to be developed
Devising a seamless pipeline of scripting, storyboarding, designing and developing VR experiences
Product pipeline for Virtual Reality
This whole process of collaborating with academia and building a product to be used by over ~200 students at the different schools of University of Michigan involved a lot of stakeholders. The vision and course material came from the faculty which was researched and aligned thoroughly by the learning experience designer to write the script of the experience. The script was then communicated to the interaction designer (Me) where I worked to convert these ideas into storyboards, 3D prototypes and made important spatial design decisions. These designs were then communicated to the VR developers for production le vel applications.
Designing the experience
After getting the script from the learning experience designer, I visualized and sketched the whole story. As a team, we went back and forth the script and filled gaps and brainstormed on new ideas. When I started prototyping for VR, I was thrown away by the lack of resources available. So I started collecting and trying out whatever material was available for rapid prototyping in VR.
SKETCHING TO IDEATE
Stage 1 - Initial ideas
After I got the script from the learning experience designer, I visualized and sketched the whole story. As a designer who is taking the first take on visualizing this virtual reality experience, I spent some time to sketch my initial ideas to get to a point where I am convinced with the first round of user flow.
The basic ideation low-fi sketches for asking right questions
SKETCHING TO COMMUNICATE
Stage 2 - Sketching 2D storyboards
As a team, we went back and forth the script, filled gaps in the flows and brainstormed on new ideas. As the ideas were more streamlined, I started working on sketches that would be used to communicate with the developer. I incorporated annotations and perspective from different angles which would help the developer save time on thinking of edge case scenarios.
Starting storyboards for chemotheraphy training being developed on Oculus Go
TAKING 2D->3D
Stage 3 - Designing 3D storyboards
Something had to be different and more about VR designs, right? When I communicated with the developers, I felt only 2D storyboards wouldn't do the job. As a designer I should be delivering things with a clearer and different perspective. I came across the blog bySaara Kamppari-Millerand Volodymyr Kurbatovfrom where I pulled different templates that bring more 360 degree & 3D perspective to the storyboards.
Templates used for calculating distances and comfortable viewing angles
3D sketches for understanding the object distances and room orientation
A STEP AHEAD
Stage 4 - Designing for body movements with video editing
Combining role play and simulation to try out the actual experience is one of the great ways to start visualizing things. Right from performing actions using controller to walking or teleporting yourself in the VR environment, conveying immersive experience in a story board is a challenge. Since I was in a remote setup, I had to design body storming results to convey it best to the developer and so, I tried my hands on in panoramic views and video editing. I put my storyboards in sequence. I used a panorama viewer to record the movement of my storyboard and then tried some effects using Adobe After Effects
Panoramic and 3D perspectives
Stage 5 - Comparing with developed versions and iterating
As a team, we decided to discuss and compare the design and final developed versions after every sprint. That provided me a check on my visualizing capabilities and what is not working out in a virtual enviroment
Storyboards being converted into actual experience in Oculus Go for Med class by Moeezo Saleem
My storyboards being converted into actual experience in Oculus Quest for Nuclear Engineering class by Moeezo Saleem
Future work
The three modules are currently being developed and the designs are being worked upon. The plan is to launch these labs in the curriculum by the end of 2020. A few of these projects are funded by Oculus at Facebook Reality Labs.
Learnings
This has been a very exciting and challenging opportunity for me. I thoroughly enjoyed designing 3D perspectives and collecting resources around how to prototype in VR.
Impact in a remote setup:
I am working fully remote for this position. Collaboration with faculty, developers and script writers has been challenging but following a definitive process has helped hugely.
Navigating in a niche field
Sketching, prototyping and designing resources in the field of XR are not very straightforward. To find the best methods to convey my designs to developers and understanding constraints of 2D and 3D prototyping has been a huge learning. In this process, I read a log of blogs, company forums and talked to experienced designers in the industry to move in the right direction.
Sketch for ideation v/s sketch for commincation
This was one of my huge learnings as a designer. Being also a developer, I find designs that can stand on their own and be self explanatory very powerful. When the designs cover all edge case scenarios, it ends up saving a lot of time for the developers and all stakeholders.
Planning for future
This process helped me find a lot resources to look at and try out for prototyping for VR. I made sure todocument my explorations.
This experience has definitely made me passionate about the future of design in AR/VR. I have also taken up teaching the course as a Student Instructor for the course Intro to AR/VR Application Design. My next step to my prototyping would be to using Unity and Unreal. I have doneprojectsin Unity and it is a great tool for designers who can code to communicate the designs with the developers
There is no limit to your imagination and its execution!
01
Team and Role
My role
Designer Developer
Team
Toshi Gupta Ruchita Lodha
Duration
8 Weeks Mar'20-May'20
Tools
Unity Vuforia ARCore Figma
02
Motivation
Dyslexia is a language based learning disorder which is found in 1 in 5 students and kids who have difficulty often avoid reading because it's stressful. We wanted to use an immersive technology like AR/VR/MR for good together with digital game based learning to enhance the learning experience and help students retain things in a less stressful and playful way.
03
Problem Statement
To build an accessible learning experience for kids diagnosed with dyslexia between the age of 3-6 years. To help kids be less stressed mentally while learning to read and write alphabets.
04
Challenges in a classroom setup
There are specific set of challenges faced by dyslexic children in a classroom setup: 1.Manual Learning: Lack of multimedia based interaction to address learnability issues. 2.Individual learning approach: Teachers are less focused on students who have dyslexia in a classroom setup. 3. Less variation in learning: Lack of useofvariety of senses such as seeing, hearing, talking, and touching.
05
Our Solution
To build an augmented reality game-based alphabet learning application for kids between age 3-6 years by augmenting the real world objects around them. The application would help students learn alphabets with phonetics and associated real world objects, identify and differentiate between alphabets and write them.
06
Why Augmented Reality?
We did a qualitative research on ways and methods adopted to help dyslexic kids learn faster like the Orton Gillingham method and the Wilson Method. We studied the online videos available for teachers to pick out the key components that are very important while helping in dyslexia. We realized thats supporting these methods with augmented reality for multi-sensory learning (Kinesthetics, Visual, Tactile, Phonetics) coupled with game based learning would help students get accurate and faster in alphabet identification
07
User Journey Mapping
Potential users: Kids who find it difficult to read, learn and identify alphabets. The idea is to build an application that augments the alphabet cards/alphabet book with 3D holograms and support the phonetic learning through sound assistance. It will have 3 main features: 1. Learning about the shape and usage of alphabet 2. Identification of alphabets after learning through games 3. Learning to draw the alphabets through guidance
08
Lo-Fi Sketches
09
Introducing JoJo - the voice assistant
To help the students engage and navigate better with the learning and games, we introduced a cute animated hippo!
The main reasons behind having a voice assistant are: 1. Help students navigate between different parts of the application 2.Lower the burden of reading instructions for dyslexic kids 3.Give a comfortable learning experience through their "Friend- JoJo"
10
Technology decisions
1. The application is developed in Unity and tested on an android and iOS mobile device 2. To enable enhancements on a book/alphabet card, we decided to go for a marker-based approach in Vuforia
*This project was carried out during COVID-19 and therefore our plan of enabling the application on Hololens could not go through.
11
High Fidelity prototype and interactions
2D menu and instructions: The game starts with a 2D menu with an introduction by a friendly assistant JoJo. JoJo guides the user to the different sections of the application like Learn, Play and draw. You can listen to JOJO's instructions in the final video attached.
Dragging/dropping and success/failure animations: To test students' capability to identify the learned alphabet, they are asked to drag and drop the associated 3D hologram into the ring options. Based on the answer the application reacts gently with animations of right and wrong answer.
Shooting with a ball: For letter identification, the application is supported with physics enabled bowling game. The students' are asked to shoot the associated hologram to the right letter glasses. The right and wrong answer are distinguished with sounds.
Drawing: The students can reference the drawing technique animation and try drawing with a physical marker.
.png)
.png)
.gif)
.gif)
.gif)
.gif)
.gif)
.gif)
.gif)
.gif)
.gif)