Guitar Instruction App

As part of the Media Arts and Sciences PhD process, I was required to take a comprehensive exam to apply principles learned in classes and my research in a new context.  At the time, I was actively taking guitar lessons and thinking about how to structure interactive physical therapy protocols for stroke patients.  The exam required me to analyze an existing guitar instruction app (Rock Prodigy) in the context of traditional approaches to musical instrument instruction as well as adaptive motor learning.  I created conceptual prototypes in Processing and proposed a complete redesign of the educational experience that would take the strengths of the app’s current interactive experience and integrate constructivist learning concepts for holistic, long-term learning.

Role: Lead Experience Designer
Contributions:

• Contextual research
• Experience and system architecture design
• Interactive conceptual prototypes built in Processing
• Presentation and documentation of final design

Dates:  December 2012 – January 2013

Backstory

Rock Prodigy (circa 2013) was an interactive guitar tutor for iPad that took many cues from Guitar Hero and Rock Band, except the input was an actual guitar and the scrolling notes were the same as seen in traditional guitar tab notation.  It featured some lessons of various styles that were loosely structured to scale in difficulty.

My comprehensive exam committee tasked me with analyzing the current experience of Rock Prodigy and other interactive teaching tools in order to design an improved interactive guitar tutor that integrated motor learning as well as other models for learning. At the time, I was about two years into learning the guitar with an instructor myself. The committee gave me four weeks to generate interactive prototypes, along with documentation and a presentation that demonstrated a challenge for the current market that would appeal to companies.

Problem

While I found that Rock Prodigy featured a promising, if loose, experience form, there were crucial issues:

1. Short-term and long-term experience disconnect: Rock Prodigy provided two experiences.  The first and primary experience was through the app.  The second experience was an online profile that showed a suggested daily lesson progression as well as a bar chart with information regarding each day's use. Functionally, there was no way for the user to access the lesson progression and bar chart through the app. But more importantly, it was unclear what the bar chart signified and how it connected to the day-to-day experience.
2. Isolated lesson experience: The app featured lessons of various types (scales, chords, etc.) which were scaled fairly well in terms of difficulty and an overall learning progression.  However, these were completely disconnected from songs.  In other words, it was up to the user to buy songs separately and practice on their own.  Therefore, technique lessons are practiced in isolation, creating a missed opportunity to connect technique to songs.
3. Lack of adaptation: The app was able to register correct and incorrect notes and therefore could provide a basic summary of lesson success (measured by percentage of correct notes).  But this information was only used in the context of the individual lesson and required interpretation by the user.  It did not provide any feedback on what lessons should be chosen next or repeated.
4. No body posture or finger position information: Rock Prodigy featured a narrator that would minimally discuss body posture for playing the guitar but no visuals were provided to demonstrate.  This was especially lacking for finger position on a fret board, which can be crucial for efficient technique.

Process

First, I completed an intensive review of related interdisciplinary fields.  I interviewed multiple instructors of varying experience and training backgrounds to learn about their process as a performer and instructor, including how they structure practice sessions for their students as well as themselves.  I also reviewed research on traditional methods of musical instrument instruction (Dalcroze, Kodaly, Orff and Suzuki) as well as models for learning and motivation.  Finally, I assessed the landscape of current technological tools that exist for guitar performers, such as interactive lessons, video instruction, and tablature tools.

Given my experience in designing systems for stroke rehabilitation, I also delved into researching how motor learning research may connect with musical instrument instruction, and how each area may inform the other.  This included a detailed review of my current approaches to interactive neurorehabilitation design and how adaptation decisions to a training protocol could be made (both by a physical therapist and an automated computer).

From this research, coupled with the previously identified problems, I selected a key application space: the adult student. I saw there was a need to address adult users who were looking to learn how to play an instrument.  Many of the traditional approaches to instruction assume a very young student who is essentially starting from scratch, learning their first instrument.  However, adults have crucially different needs and expectations and therefore require different instructional approaches.

Therefore my main goal was to extend the Rock Prodigy instructional and experience design to utilize a learning hierarchy based upon a particular song so that a user could learn guitar by immediately diving into music that is important to them all while connecting component based learning with songs.

Technique Components

The lowest level of my learning model identified how a song can be defined by Guitar Fundamentals (Posture, Picking, etc), Notes (Octaves, Scales), Chords and Style (Slides, Hammer Ons, Pull Offs, etc).  Therefore, any song could be broken down into components to inform the structure of future lessons, not only in terms of what material to introduce, but also a priority of the order in which material should be introduced.

Sequencing Components

Overall, a structure was imposed that Notes material should be introduced before Chords material, which in turn, should be introduced before Style material.  However, the progression should not be explicitly linear, in that not all Note material needs to be introduced before moving on to Chords. 

A learning hierarchy should be established early on with the user as to how notes build to chords and how style can modify both.  In addition, the hierarchy supports decomposition.  If a user is having problems with lessons in Style, then the user can jump back to Chords or Note lessons to work on components integrated in Style.  

This structure also supports the changing needs for the student over time.  Initially, a lot of practice focus needs to be on Notes level material to work on detailed components of playing a guitar.  However, as a student becomes more proficient, this low level training is less of a focus and more emphasis needs to be placed on higher-level Style training.

Sequencing Lessons

I designed the overall learning experience in blocks of seven sessions (ideally completed in a week or stretched to a longer timeline).  These seven lessons were defined into three types:

• New Topic/Review Topic (2 Sessions per week) in which new material is introduced or practiced again.
• Component Practice (4 Sessions per week) in which a subset of material from the New Topic lesson is isolated and allowed more time for focused practice.
• Critical Listening (1 Session per week) in which the user is encouraged to find other music examples to explore and critique from technical, performance and creative inspiration perspectives.

The underlying concept for this design is that lessons each week should introduce new material or review previous material, allow some time for component-level practice as well as provide opportunities for the user to critically listen to music given their current experiences with the guitar.

Each New Topic / Review Topic session was designed with six components (each estimated to take about 10 minutes):

• Warm Up – Start the user with something they have done before to build confidence.
• Lesson – New material (such as a pentatonic scale) is introduced via a narrated interactive session, very similar to the current Rock Prodigy design.
• Technique Exercises – Isolate components of the song that use the newly introduced material for practice.  (For example, if the user wants to learn Purple Haze, show where the pentatonic scale is used in the song.)
• Application – Allow the user to explore the material (such as a pentatonic scale) in another song or with a simple backing track.
• Song – Practice the goal song.
• Free Jam – Allow the user to explore anything they wish.

The goal is to help the user connect component learning with the overall form of a song.

After the completion of each session, the user is asked to record a sample of something they feel very confident playing as well as a sample of something they are still struggling to play as a form of self-documenting progress.

Updated Feedback Design

While the original Rock Prodigy design was engaging for real time use, there were a few components I proposed to add:

Real Time Feedback was redesigned to include an optional animation at the bottom of the screen that demonstrated proper finger position on the fret board.  Also, additional notation could be added for suggested picking directions.  Both of these pieces of information are important for playing a song efficiently and cleanly.

Post Lesson Feedback was a new feature I designed.  After the completion of a lesson, an interface would play back the audio from what the user performed during the lesson with highlighted tablature to show which notes were played correctly as well as incorrectly.  The goal was to connect real time feedback with a post lesson review through a listening exercise as there was previously not an opportunity to reflect on areas for correction beyond the real time experience.

Adaptive Lessons

Another goal of my new design was to provide lessons that could adapt to a user’s progress.  Keeping in alignment with the previously described model, adaptation would be crucial at multiple levels of the experience.  First, I proposed that lessons could dynamically increase or decrease the beats per minute to control the challenge.  Secondly, I proposed the design for a partially observable Markov decision process that could move between different states (lessons) based upon measured user progress and the overall model structure of the lessons.  The underlying concept was to demonstrate to the user how to self-evaluate and structure guitar practice so they could feel empowered to later structure their own practice time.

Teaching How To Practice

Practicing any musical instrument is a journey, and many times the student loses perspective on long-term progression when faced with the more immediate problems while practicing.  Using the larger metaphor of a map and a journey, I created an interface to show how far the user had come in their learning by providing two pieces of information: the quantitative evaluations made by the system as well as the audio summaries the user was asked to record along the way.  Not only can the user see from the system's evaluations how they are improving, but more personally, can hear and experience their own improvement directly from their recordings.

The end goal with the designed instructional and feedback hierarchy, as well as the computational evaluation, was to empower a user to ultimately feel comfortable to create their own lessons.  This was the primary reason for using the guitar pedal metaphor in the design.  Guitar effects pedals are inherently modular and unique in configuration to each performer.  After the user of the app has a chance to build success in learning a song and see what works for them in their learning process, the app allows the user to create their own lesson structure by choosing and connecting the lesson material however they want.

Future Extensions

In addition to the proposed design, I suggested a few avenues for future consideration:

• Embedded sensing could be incorporated with a guitar to detect additional features that may not be possible with just the audio output.
• A community could be created to allow users to share lesson plans and progress with one another as further inspiration and motivation.
• Famous guitarists could share their own learning progression to further encourage the ideas that guitar virtuosity takes time and practice.
• This app could also be used between existing instructor and student relationships where the instructor could create lesson material and track the progression of a student’s practice time.

Outcome

After completing the conceptual prototypes, I conducted evaluation interviews of the prototypes with two people I had interviewed at the start of research as well as one person completely new to the project.  

Overall, the lesson structure was viewed as a novel approach to teaching.  There was some concern that the rigidity of the lesson structure would be a problem if a student is really struggling, but acknowledged that the adaptive lessons would help.

Both the critical listening and post-song feedback were identified as strong components to the experience. It was expressed that finding influences and related music is an important factor to exploring music and moving beyond learning one specific song. There was some concern about the motivations for the user to engage with this component and to avoid it coming across as “busy work.”

The geographic metaphor of the map was viewed as a little confusing as it could be argued there is one efficient path between two locations on a map, as opposed to the many ways a user might progress through learning the guitar. One suggestion was to use a fictional map where regions, or states, were divided by different lesson material. However, the audio summary featured in the map and the requirement to establish a routine schedule were both viewed as strong features for engagement.

In addition to these few reviews, I proposed additional metrics for future testing to measure user engagement and learning including usage statistics of time spent with various aspects of the app and a self-assessment questionnaire. I also suggested a small user study working with an instructor to verify if the adaptable lessons are making beneficial choices.

The proposed re-design of Rock Prodigy was well received by my comprehensive exam committee and I passed without revisions.  Their feedback echoed the feedback received in evaluating the prototypes.  They further pushed me to refine the visual composition of my prototypes and ensure the content did not take away from the overall experience.

Reflection

This was my first opportunity to take my development as an experience designer and apply it in a completely new context in a very condensed amount of time, on my own. While I was somewhat familiar with guitar instruction and had some starting thoughts about possible innovation in this space, I had to perform efficient contextual research and produce enough prototypes of sufficient interactive fidelity to demonstrate the model and learning experience.  As a result, I found that this is the work I love to do: dive into a new context and find my way out, prototyping along the way.

Looking back on this work now, the visual content and forms were not where I would like them to be.  At the time, I had to make a decision that the biggest innovation was the learning model and experience, so I created prototypes to show those pieces. However, if I was to work on this project again, I would want to strengthen the visual content as well as establish a more uniform visual esthetic.  The prototypes I did create represented, at the time, some of the most substantial work I had done in Processing and as a result I was able to add that language to my design toolkit.