Milestone # 2 - 11/20/2024

Sprint

In Milestone 2, we further fleshed out how single-player and multiplayer would work. We decided to have a training/single-player mode where the player could learn each letter while casting spells. A timer will count down while the user tries to cast as many specific spells as possible. Multiplayer mode would involve having players sign certain letters to cast spells against opponents. Players can also cast counter spells to stop attacks (for example, a fire spell can be cancelled out by a water spell). For multiplayer, we decided that passthrough would not be the best way to implement the game, therefore we pivoted to making the game 100% VR. Since there will no longer be a physical environment, we must figure out the challenges of developing multiplayer with avatars and a virtual environment.

In this milestone we focused on implementing basic functionality to single-player.

Sprint Overview

• Implemented left and right hand gestures for all ASL letters.

• ​Start menu and gesture detection panel added for single-player mode.

• Mock spells now fire from the opposite (non-signing) hand's palm when a specific letter is gestured​. This allows for players to aim where their spells are cast.

• Letters are now grouped into three spell types (fire, water, earth) and are randomized every complete cycle.

Progress

• Hand gestures: Left and right-hand gestures are implemented except for letters that involve hand movement (J and Z). We decided to use Meta's hand tracking SDK instead of OpenXR's XR Hands because the latter was having issues with customizing hand gestures. With Meta's hand-tracking, there are some quirks to fix with similar letters such as P vs Q that sometimes get mixed up. Also, a UI panel with all 26 letters is displayed to the user, so they can learn letters as they practice coordinating their spell direction and element. ​

  • For precision purposes for classifying certain ASL letters, we decided to have players sign towards themselves rather than sign outwards (as in to the person they are communicating with). Many of the ASL letters look nearly identical if you are not looking at the palm-facing part of the gesture (for example, M and N). Thus, for the hand gesture classification to work, all fingers in the gesture must be visible, which means having users sign facing themselves. ​

• Spell casting: Red, blue, and green spheres now shoot out of the player's left palm when a particular letter is detected. The UI board shows the player which letters to sign to cast either fire, water, or earth. When a particular element/letter is signed, the letter is deleted from the board. When the user signs all the displayed letters on the board, the letters are reshuffled to represent different spells. This ensures that players don't just memorize which signs are high power spells and are forced to learn every letter.