VR Hackathon - Vox Augment

Winner of best technical achievement at 2017 Seattle VR Hackathon. VR is a medium that’s quickly becoming a new creative outlet. We believe it is a way of making music based on natural movements, without relying on existing instrument interfaces. The level of immersion can unlock latent musical abilities in new users, or extend the abilities of existing musicians.

Summary: Vox Augmento represents an attempt to leverage VR technology for creative musical expression. What are the advantages of interacting with holograms musically? This experience allows you to easily create your musical interface in real time. with a three-dimensional paintbrush and the ultimate synthesizer– your voice. As you paint in space, audio is recorded and mapped to that brushstroke.

Inspiration

Phillip proposed a general goal of creating a unique VR-centric music tool. Our initial analogue was a theremin, but in VR. The team has many musicians and individuals enthusiastic for novel expression in VR. Currently many other music or sonic manipulation apps rely heavily on skeuomorphic principles. Our design question was: How could we build a musical instrument that could not be realized outside VR.

Philosophy

We built Vox Augmento to enable both novice and experienced musical enthusiasts to express their musicality. The barrier to entry is low but the instrument’s full expressive capability will need practice and mastery just like a real instrument.

Story for Vox Augment

Sonic interaction and manipulation Music and sound involves interaction and movement. The most engaged musical players often move in ways that don’t directly influence the sound being produced. Vox Augmento makes fluid bodily expression inseparable from musical expression. Providing a novel mode of expression was high on the priority list for users

Visual understanding Users want to revisit their actions in VR space, know that what they do exists in space.

Experimentation and discovery Finger painting is wonderful. Playing, creating, and destroying is important to the user. Even during initial testing, unexpected expressive usages were discovered. By purposely minimizing menus and modes, a user is more free to try things for themselves.

Mastery Novices in music can often feel that an instrument is daunting and be discouraged. The mode of expression should allow for both novices and experts to create amazing experiences. The instrument makes its limits and functions easily discoverable. Users who want to achieve mastery to express themselves in more fluid ways are encouraged.

Product

Journey

MVP A user can paint what they speak as if placing a waveform in air. Moving the controller higher raises pitch, lowers the controller lowers the pitch. Colliding with the painted waveform will replay the sound.

Discovery Team members discussed concepts of how sound and music interact within VR space. Quickly working through how people of different experience levels try to make music. Then asked how to best represent interaction with sounds or music. What does the X, Y, Z access mean to the creation process? Initial analogs to real world editing were demonstrated using Abelton. This let the entire team understand basic concepts of editing quick sound samples.

A key insight was that not all users were musically literate. Would it be possible to use this VR as a medium to unlock a new world of creativity.

The team determined which qualities overlapped between different experiences, and how to guide novices to mastery without being overwhelming.

Example concepts

• A particle field representing the playhead
• Moving, rotating, shaping allow for changes in sound
• Striking a sound plays that sound once over
• Sounds correspond to an existing model

Design

Assets were created in Tilt Brush, Blocks, and Maya.

We conducted quick surveys asking what comes to mind when trying to make a sound loop. This revealed several answers, many responses involved drawing a circular loop to represent make a sound play over-and-over. Another response was to let any sound loop, this would be achieved by selecting the sound for a select amount of time.

For ease MVP, a menu system or multi tap would allow loop mechanic to work.

Visuals Setting

Users are on a rock landscape platform. Burning man on the cliff. Controller - A bone musical wand is used to represent an ancient paint brush. A light trail flows the tip. Output - White sound bubbles are generated from the controller to represent record sound.

Interaction Controller Trigger

Record spoken sounds and paints out the sounds as bubbles in the VR world Modulation - Recording sounds while moving the controller in different motions will change the sound on playback Collision - Collide the controller into a sound bubble to play it back Delete - Menu button clears all record sonic recordings

Development

Unity - Visual engine, environment, Max/MSP - Realtime audio modulation and effects

First steps - Realtime link between Unity and Max. Get data about controller and collisions into the audio engine. Crucial to get this step to work, otherwise new platform would need to be chosen.

Second steps - Parallel development to produce working assets and interaction. Recording and modulation Mapping controller movements Tweaking control schemes and sensitivities

Third - Polish and final touches

Audio cleanup, looping, multi controller ideas were non-crucial and backlogged as nice to have. These were not accomplished during the initial build.

Persona There was a man who like to sing. This man liked to dance. One day he wanted to dance to sing! This man may not have been musically literate, but with VR he would more quickly move up to proficiency.

Basic Mechanic Users using the headset microphone and controller paint sounds they make into the VR environment. The shape of the painting alters/modulates the sound. The painted sounds can be interacted with to re-play them. Painting sounds into a circular loop will let the painted sound loop over and over.

Future or stretch goal Mechanics

• Advanced control over the sounds in VR space. This includes editing the sounds, replacing, looping, and duplicating. Granular control will allow for more complex expression, ever increasing the range for users who try to achieve mastery.
• We will avoid making digital representations of real world instruments. No drums, guitars, saxophones, etc…

Demo’d Product

Users are handed one controller with little instruction beyond the intent of the app. The goal of the user interaction is about discovery. The usage and discovery is not something the developers would be able to ever fully predict. The demo may even evoke new usages and practices not envisioned.