Postcards
POSTCARDS: EXPRESSIVE MUSIC AND ADVANCED PERFORMANCE TECHNIQUES FOR WII REMOTE CONTROLLERS by Parker Bert, Patrick Hart, and Gabriel Vigliensoni
ABSTRACT
The goal of our project was to create expressive, developed music using Wii remotes. To do this, we produced the following; a simple and adaptable software interface that exploits the strengths of the Wii remotes for musical performance, a notation style that can be easily explained to new performers, an explanation of basic technique fundamentals so that performers can achieve competency with the instrument, a starter set of sounds designed with Wii remote performance in mind, and a collection of short compositions that explore some of the possibilities of performance with Wii remotes.
1. INTRODUCTION
In order to achieve expert interaction by means of the use of gestural input devices to control real-time sound synthesis, Wanderley suggested in that the process will be divided
in four parts:
• Definition and typologies of gesture
• Gesture acquisition and input device design
• Mapping of gestural variables to synthesis variables
• Synthesis algorithms
Since the input device was pre-designed, during the course of our research we focused on creating gestures, mappings and synthesis based on the sensing possibilities of the remotes. We continually refined all aspects of the process as we worked on composition and performance techniques. Each one of the authors of this work brought their specific expertise to the project. Chapter 2, by Gabriel Vigliensoni, covers the features and design of the Wii remote controllers; what it senses and how, and how we can retrieve and utilize that information. Chapter 3, by Patrick Hart, explores compositional issues of writing for Wii remotes, from the sound palettes, including mapping and synthesis algorithms, to notational issues. Chapter 4, by Parker Bert, deals with aspects of technique, pedagogy, effective practice and rehearsal, and expressive performance for our musical instruments. In chapter 5 we will finish our paper with our thoughts and reflections about control subtleties and expression using Wii remote controllers, what constraints they have for musical composition and performance and how musical expression can be improved with further development of Wii remote controllers.
2. INPUT DEVICE STUDY AND FRAMEWORK PIPELINE DESIGN
In our technical research, we focused on how to control sound synthesis parameters by means of a pre-existing gestural controller developed for the video-game industry, the
Nintendo Wii Remote, also known as the wiimote. Although it was not conceived as a Digital Musical Instrument (DMI)—in which the gestural controller unit can be separated from
the sound generation unit—we wanted to test if the device had enough reliable possibilities in the acquisition of musical gestures to be used as a serious input device for the control of sound synthesis parameters via expert interaction. It is important to note that in our research we looked for the best gestures to perform with the input device as it is already
designed.
2.1. The Wii remote controller
The wiimote was launched in 2006 in conjunction with the Nintendo Wii console. Since its release it has received much attention due to its novel opportunities for interaction in a gaming context. In music and visual performance, it has been the focus of experimentation for many DJs and VJs, who have used different software frameworks to acquire and map the sensed data to control different audio-visual variables in their shows [11]. Their main attractions have been the range of control offered by the many embedded sensors and buttons, and its wireless capabilities that transmit data through a popular protocol. All of this exists at an affordable price due to a large scale economy where more than 77 million units have been sold worldwide. However, the official specifications of the device are unpublished, so researchers, developers, and musicians have had to rely on their own experimentation and the informal knowledge provided by the global hacking community that has collectively reverse-engineered their internal working. As posed in [6], sensors are the sense organs of a machine, converting physical energy into electricity, linking the outside world to the machine world. Thus, we will review the main technical aspects of the sensors and actuators involved in the design of the Wii remote controller.
2.2. Wii Remote’s input features. Much of the following information has been collected through web communities, Wiibrew being the largest one, which is a site mainly devoted to producing hardware and software for the Wii console for purposes outside those intended by Nintendo.
The wiimote has a three-axis accelerometer, an IR Camera with an object tracking processor, 12 buttons, and an expansion port allowing external input features of extension controllers such as the Classic Controller and the Nunchuck.
2.2.1. Buttons
The Wii remote controller has 11 buttons on its front face, and one trigger-style button on the back. Most of these are membrane switches, while the others are microswitch click
buttons. All of them except for the power button report a 1-bit if pressed or a 0-bit otherwise, which by default is sent only when the state of any button changes. Button pressing is reported as bits in a two-byte message.
2.2.2. Accelerometer
The wiimote’s accelerometer is the Analog Devices’ ADXL330. It is a small, thin, low power, three-axis accelerometer with signal conditioning voltage outputs embedded in a single integrated circuit. The system measures the X, Y, and Z linear acceleration of a small proof mass enclosed in a single structure. It can be used to measure dynamic acceleration resulting from motion, shock or vibration, as well as static acceleration of gravity in tilt sensing applications, over a range of +/- 3g. Although the voltage range of operation goes from 2.0V to 3.6V, its output is ratiometric, so the output sensitivity varies proportionally to the supply voltage. According to the specification sheet, its frequency response for the X and Y axis is 1600Hz, and for the Z axis is 550Hz. However, we do not know the actual frequency response because it is controlled by capacitors whose values are not published, that smooth the signal. In normal reporting mode, the acceleration data is reported as three consecutive bytes, for X, Y, and Z axis respectively. Figure 1 shows how the system of coordinates is set in the controller.
Figure 1. Coordinate system used by the Wii Remote (figure taken from [1])
2.2.3. IR Camera
The wiimote comes with a monochrome IR camera manufactured by PixArt Imaging. It has a resolution of 128 * 96 pixels with built-in processing, allowing a maximum resolution of 1024*768 pixels using 8x subpixel analysis. Moreover, the onboard chip allows multitracking of up to four simultaneous IR light sources, returning their approximate center point position and size over the bluetooth connection. This last characteristic is very important because hardware-based tracking helps to reduce the CPU power needed to perform real-time computer-vision tracking. The field of view of the camera is about 45 degrees horizontally and vertically, which is relatively narrow. The wiimote’s preferred peak wavelength is set up at 940nm, being five times more sensitive than the consumer-common wavelength of 850nm [7]. The sensor bar that comes with the Wii console has two arrays of four LEDs each, which are tracked by the IR camera to provide pointing information. In basic mode, the camera returns 10 bytes of data corresponding to
the X and Y locations of the four blobs.
2.3. Wii Remote’s feedback features
The Wii remote provides vibrotactile, sonic and visual feedback. The visual feedback is given by 4 LEDs noting the number of device being held, from 1 to 4. A small motor driven by the Wii remote controller at 3.3.VDC attached to an off-center Weight provides vibrotactile feedback. The model of the motor has been changing over time, so there are many variations. Finally, a piezoelectric lo-quality speaker is used to provide simple sonic
feedback. By default, the standard values for the transmitted digital signal are a sample rate of 3000Hz and a bit-depth of 4, with Pulse Code Modulation (PCM) coding.
2.4. Wii Motion Plus extension
The Wii Motion Plus extension was released in 2009. It can be securely connected to the wiimote by means of the 6-pin expansion port at the bottom of the controller. Thanks to
Micro-Electromechanical Systems (MEMS) technology—which allows the production of extremely small mechanical structures using semiconductor technology [8]—the Motion plus contains two gyro sensors; a dual-axis Invensense IDG-600 (currently IDG-650) for pitch and roll, and a single-axis Epson Toyocom X3500W for yaw. When the sensor is ro-
tated about the X, Y or Z axis, the Coriolis Effect causes the mass proof to vibrate in-plane. This vibration is detected by measuring the capacitance change between the mechanical structure and fixed electrodes. The resulting signal is amplified, demodulated, and filtered to produce an analog voltage that is proportional to the angular rate. As the wiimote is intended for gaming, the controller needs to measure quick movements, thus the device’s gyroscopes are designed to detect a rotation of up to 1600 degrees per second. However, for precise and accurate situations it needs to measure slow movements as well, so there are two modes of sending data: one for fast and the other for slow movements. A mode is automatically enabled depending on the kind of data being sensed. Thus, by means of the two gyroscopes’ information, the device can report angular rate in all three axis, allowing full orientation tracking. To overcome the problem of temperature drifting, a software-based solution was implemented in its design. Although the communication between the extension and the main controller is bidirectional synchronous serial, it is encrypted; thus, the protocol is still unknown.
Hence combining the accelerometer and the gyroscopes’ sensing information, the Wii remote can detect movement along 6 different axis, as we showed in figure 1. Thus, not
only movements along straight lines can be detected, but also rotation and twisting movements. Figure 2 shows how different a series of eighth notes using pitch, yaw, and roll strokes is measured by the wiimote’s gyroscopes and accelerometers. It can be seen that the data of each axis acquired by the former is more orthogonal than that acquired
by the latter, which is more noisy. Furthermore, it can be seen that in the case of acceleration measurement, the 3-axis are correlated in some way, being difficult to play strokes in one direction without affecting the other axis.
Figure 2. Comparison of how eighth notes using pitch, yaw and roll strokes are measured by the Wii remote’s accelerom-
eters and gyroscopes. Note the highly uncorrelated data measured by the gyroscopes for each axis. On the contrary, the
accelerometers measure highly correlated data for each one of the strokes in all of the axis
2.5. Wii Remote’s wireless communication
The Wii remote uses standard Bluetooth technology to communicate with the Wii console. It makes use of the Bluetooth Human Device Interface (HDI) protocol and the Broadcom BCM2042 bluetooth system-on-a-chip. By this means, it appears as a standard bluetooth device to any host and can be synced without any authentication, which is uncommon
for bluetooth devices. However, it does not use the standard data types and HDI descriptors, so the actual contents appear undefined with standard HDI drivers. There are a number of different data reporting modes from the Wii remote controller to the host through the Bluetooth protocol, depending on the external peripherals being used and the mode selected. As we will see later, we will make use of the core buttons, the accelerometers, and the gyroscopes embedded in the extension controller; so a typical data reporting structure should have 2 modes and 21 data bytes:
(a1) 35 BB BB AA AA AA EE EE EE EE EE…
… EE EE EE EE EE EE EE EE EE EE EE
BB being the data from the buttons, AA the acceleration, and EE the data from the Motion Plus’ gyroscopes.
2.6. Wii remote’s data transmission and remapping
As the Wii Remote’s communication to the host is based in the popular Bluetooth protocol, there have been efforts in developing software to acquire the data and reuse it on other platforms.
The Glove Programmable Input Emulator (GlovePie), is a non-commercial application that provides support for input devices in Microsoft Windows, and has been extended to support the Wii Remote and their expansion controllers (as well as the Essential Reality P5 Glove, Sony Dual Shock 3, NaturalPoint TrackIR, all keyboards, joysticks, gamepads and mice supported by Windows, Polhemus and Ascension trackers, among others) [9]. Although it has been extensively used by a large community of developers because it provides low-level access to data and is well documented, it does not run in OS X.
junXion is STEIM’s OS X data routing commercial application allowing users to acquire data from a number of human input devices (HID), process and remap it, with MIDI and Open Sound Control (OSC) events as its output. Although junXion’s last version v4 supports the Wii Remote’s accelerometers, buttons, and IR data, it does not support the Wii Motion Plus’ gyroscopes’ data.
Using the C Darwin Remote library to read and send data to the Wii remote controller, Nunchuck and Classic controllers, Masayuki Akamatsu developed the non-commercial and popular aka.wiiremote Max/MSP object at IAMAS. It can run on Windows and OS X platforms and the source code is also distributed for further development. This Max/MSP external supports all wiimote’s buttons, acceleration, and IR information, but does not support the gyroscope’s data.
OSCulator is a low-price (U$19) commercial application developed for OS X that links novel input devices and external controllers (such as the IPad, IPhone, Lemur, Wii remotes, Wii Motion PlusSpace Navigator, Wacom tablets, HID Joystick emulator, as well as many computer mice and keyboards) to music and video applications. OSCulator provides the possibility to acquire the information of up to 8 Wii remote controllers at the same time and smooth their data with a non explicitly declared function (it only provides a slider whose range can vary from 0 to 100) for each variable. The software returns all data it collects normalized between -1 and 1 floating point numbers. It also provides convenient visualization tool called quick look that allows to see the evolution over time of each sensed variable, as we showed in figure 2. Finally, OSCulator can map and duplicate, if desired, any input variable to any target devices or applications using the MIDI or OSC protocols. Figure 3 shows our project’s mapping from the Wii remote controller No 1 to any application in the same computer receiving OSC.
Figure 3. Mapping Wii remote buttons, accelerometer, as well as gyroscope data, to any another application in the same computer using the OpenSoundControl capabilities of
the OSCulator software.
2.7. Project Framework
The main goal of our project was to create expressive, developed music using the Wii remotes controllers as input devices. To achieve this, we studied the previously designed controllers to understand and test how they work, what they sense, and how we should integrate them with other applications to develop an easy-to-use, musically focused, flexible and reliable framework. With this background information we could select which sensors would be most appropriate to acquire a performer’s gestures and map their values to any synthesis parameters.
2.7.1. Sensor selection
As we have seen, the gyroscopes embedded in the Motion Plus can sense close to orthogonal angular rate variables. If this information is used in companion with the 3-axis acceleration sensed by the Wii Remote we could measure six degrees of freedom (DOF) in a relative position. If we would like to measure an absolute position, the IR camera with the
sensor bar would be helpful in order to have a permanent point of reference. However, as we wanted to have absolute freedom of movement, we opted to use the device in a relative context. Hence, the six data streams given by the measurement of the acceleration and angular rate could be used as flux of information to control continuous parameters. On the other hand, the large amount of buttons could be used to trigger events or use them to change the status of any preset situation. However, for our project we did not use all the possible variables, but rather selected some of them.
Figure 4 shows which parameters of each wiimote device we used in our setup.
Figure 4. Mapping of each one of the Wii remote’s selected sensors to trigger notes or control synthesis parameters (from OSC to MIDI data)
2.7.2. Framework pipeline
The framework pipeline was defined by using OSCulator to receive and normalize the raw data from the controllers via the Bluetooth standard and to transmit it to the OSC protocol. Max/MSP should receive, condition and map the information to MIDI notes and continuous controllers. Finally, Logic’s synthesizers opened in Apple’s Mainstage would be the main sound synthesis engine. Figure 5 shows a simplified version of the setup.
2.7.3. OSC messages, conditioning, and routing
Using OSCulator, we routed the button and sensor’s raw data for each one of the four Wii remotes to the same computer using the generic hostname localhost. Max/MSP allows the reception of OSC very straightforward using the udpreceive object. However, in order to easily route messages using URL-style addresses, the OSC-route object from CNMAT is required. We conditioned the signal of the accelerometer by calculating the median, and scaled the results to be used afterwards as a MIDI NoteOn Velocity, as we had decided previously.
Figure 5. Simplified setup pipeline for the creation of music using wiimotes
2.8. Making and Controlling Sounds
As we posed in figure 4, we defined the use of the Wii remote controllers in three basic operations: one for triggering notes, another to modulate the sounds’ timbre, and the last
to control the overall behaviour of the system without making any sound.
2.8.1. Triggering Logic
We thought about the triggering mode as something similar to a drum stroke. When a hit is made, a sound must be triggered. In addition, when any of the D-Pad buttons are pressed, the pitch of the sound must be changed. To achieve a clear triggering, we analyze the data extracted from the accelerometer and the two gyroscopes graphed in figure 2. We decided that the pitch and yaw angular rate measuring was sufficiently uncorrelated to other variables to be used as a trigger signal. We tried some triggering levels, and decided that the threshold should be around 0.5 in a scale from 0 to 1. To obtain the mapping posed in figure 4, we developed a logic that allowed the angular rate for pitch or yaw to trigger a note when each one of them trespassed a sensitivity limit. Although this parameter started as a fixed value, we tweaked it depending on how the performer felt the trigger threshold.
2.8.2. Timbre modulation Logic
To allow control of the timbre without unwanted triggered
notes, we decided that when A or B buttons were pressed, no
MIDI notes would be triggered and the angular rate would
control two different MIDI continuous controllers (CC). This
would allow us to change the behaviour of the controller
from a drum stick to more of a timbre-exploration device.
We achieve this goal by using simple gate objects.
2.8.3. Program changes and MIDI Off Logic
The last two important developments of the technical as-
pects of our framework were to find a way to automate the
changing of all MIDI note numbers assigned to the D-Pad
for each patch, and to find a way to mute all kinds of MIDI
information being sent by the controllers to the synthesizers
in the whole environment. By this means, we obviated hav-
ing a computer screen on-stage, by being able to change pa-
rameters in between pieces directly from the remotes with-
out triggering unwanted notes. We developed an implemen-
tation for changing and sending all MIDI Note Numbers for
each Wii remote in the overall environment that is easy to
customize for future use.
2.8.4. User Panel
The user panel was designed to provide clear visual feed-
back for the patch being played, the active controllers, and
the reception of OSC data. It also allows easy access to all
major subpatches for each one of the Wii remote controllers.
The main patch also contains, in edit mode, all information
regarding the MIDI notes for each one of the patches, the
MIDI Port and channels for each one of the wiimotes, and
the logic to acquire and transmit the program changes to
Mainstage, the next application in the framework pipeline.
3. COMPOSITION
For this project, with regard to both the sounds and the mu-
sic, we thought it was important that we created something
artistically viable that could be taken seriously in an aca-
demic setting while remaining accessible and encouraging
the largest number of people to undertake similar work with
Wii remotes. To this end, we wanted to create a piece that
stood on its own but also showcased the particular talents
of the remotes. Postcards is a roughly 15-minute long piece
made up of seven short pieces for four Wii remotes and two
performers, with spoken text in between each movement.
The text is a narrative account of a vacation gone awry, and
for the premiere performance Gabriel processed the text live
with the Soundcatcher [14], a hardware device that he built.
The live processing enabled us to create a more immersive
atmosphere for the dialogue and also helped to prepare for
each of seven different sound sets for the Wii remotes. Hav-
ing a narrative to tie all of the short pieces together helped
to distinguish the pieces from merely a set of etudes or prac-
tice studies. Nevertheless, each short piece explores differ-
ent territory through both the types of gestures used and the
sound synthesis options peculiar to each section, and each
serves specific pedagogical functions.
3.1. Compositional strenghts of the Wii remote controllers
When creating the mappings for the Wii remotes that we
eventually settled on, we wanted to focus on two partic-
ular strengths of the remotes: the ability to produce dis-
tinct rhythms with precision and the ability to send a va-
riety of streams of control data that could be used to effect
the sound. Our initial inclination was to combine these two
ideas; this resulted in the problem that percussive motions
ruin the smooth stream of control data, creating unwanted
”hiccups” in the sound. We attempted to smooth the control
data using smoothing options in OSCulator, but smoothing
the data enough to avoid spikes generated by percussive mo-
tions introduced sufficient latency to make the control data
largely useless. Therefore, we turned our focus to devel-
oping a system that switched between triggering midi notes
and sending control data with ease. We divided the contin-
uous data sent by the Wii remotes (excluding IR data) into
two basic categories: that generated by velocity (accelerom-
eter data) and that generated by moving the remotes to a spe-
cific position (gyroscope data). We decided to assign each
type to one of the two most easily-accessible buttons on the
Wii remote. The end result is that holding the B button tem-
porarily halts midi note generation and allows the performer
to modify the sound with the velocity of his or her gestures,
and holding the A button similarly halts midi note gener-
ation but allows the performer to modify the sound based
on the position of the Wii remotes. The only control data
that we preserved in absence of conditional button presses
is a mapping of pitch to velocity. This allows the performer
to play loud notes high in the air and soft notes low to the
ground, and since velocity changes make no difference once
a midi note has been triggered, it is not susceptible to the
problems of ”sound hiccups.” We also discovered an inter-
esting unintended feature, which is that when sending infor-
mation with the B button held down, pressing the A button
freezes the velocity value until the A or B button is released.
This permits swells into a loud sustained sound, as demon-
strated in the right hand of player one in measure 12 of Post-
card No 3.
Although the ability to send control data offers clear ad-
vantages to acoustic percussion instruments, we wanted the
capabilities for triggering notes to be competitive as well.
Using the Wii Motion Plus allowed us to trigger different
midi notes based on accelerations in both directions on the
X, Y and Z axis. We settled on each remote having a primary
strike (on the Y axis) and opposing secondary strikes (on the
X axis), as this seemed to be the most natural and comfort-
able. The directional pad, as the most accessible set of but-
tons after the A and B buttons (which were already in use),
is used to change the midi notes being triggered by primary
and side strikes. After a few months of using the Wii re-
motes, it is clear that a virtuoso performer using our playing
techniques (which we will touch upon later in the paper) as a
foundation could achieve impressive results. The most prob-
lematic aspect of these mappings is that the directional pad
is very difficult to use, both from an ergonomic standpoint
and a reliability standpoint, as the pad is quite small and ap-
plying too much pressure can cause the depressed direction
to change. However, we decided that the other remaining
buttons were too inaccessible to be used in this fashion.
Our arrival at a collection of short pieces was largely fu-
eled by a desire to explore as many different sound synthesis
options with the Wii remotes as possible. Each piece uses
a different set of sounds and requires different performance
techniques to be played successfully, lending the piece some
value from a pedagogical standpoint as well. In keeping
with our goal of making composition for Wii remotes more
accessible, we used only stock Logic Studio synthesizers
and effects for sound synthesis, enabling anyone with the
program to use the complete set of sounds. Several impor-
tant problems arose when developing our sound sets. First,
and perhaps most importantly, the midi notes triggered by
strikes of the Wii remotes are generated by a ”makenote”
object, which means that they are all (by default) the same
velocity and duration. As a result, we needed to be creative
with how we might produce long sounds and short sounds
alike with the same synthesis engine. (It is worth noting
that we decided not to use midi sustain pedals, but that is
a viable avenue to explore in the future.) Different pieces
attack the problem in different ways; player two in Postcard
2 increases the effective length of notes in the right hand by
manipulating filter resonance and cutoff with roll in the left
hand, and both players can increase the mix of a delay with
velocity, also lengthening sounds; in Postcard 3 player one
can dramatically change the intensity and decay type of the
right hand with the B-A hack, which is mapped to a battery
of seven different synth parameters; Postcards No 4 and No 5
both have delays which can be activated with control data;
lastly, postcards 6 and 7 use the effective but decidedly un-
technological approach of having long sounds mapped to
some remotes and short sounds mapped to others. We spent
considerably more energy figuring out how to lengthen short
sounds than shortening long sounds. This was occasion-
ally problematic, particularly in the early stages of learn-
ing the music. In Postcard No 3, for example, accidental
triggers by the right hand of player one could quickly be-
come overwhelming, requiring a few seconds of ”cool-down
time” until the multi-second decay of the synth had expired.
Of course, there is nothing preventing anyone from control-
ling parameters which reduce the intensity and duration of
a sound; this simply wasn’t the approach that we typically
took.
3.2. Sound synthesis methods in Postcards
Postcard No 1 is a simple rhythmic study intended to set a
pleasant atmosphere (that will be ruined in short order!) for
the piece. Each remote is controlling similar patches in ES2.
The sounds are short and precise and the harmonic language
is very simple. Postcard No 2 is considerably more unusual,
and the sounds are more complex to accompany the first
sign of trouble in the narration. Control data is used for the
first time in this piece; each player can control a delay and
several pitch shifters with the B buttons of their remotes,
and player two can control filter cutoff and resonance for
the right hand with the A button on the left hand. Both of
the left hand remotes have a lengthy sound which slowly
either rises or falls, and the right hand remotes control a
shorter, inharmonic, percussive sound. The velocity of the
right hand remotes is mapped to pitch, so the tremolo ges-
tures produce a downward glissando effect. Postcard No 3 is
a dark, ethereal piece, punctuated by tremendous booms by
the left hand of performer one. These booms are produced
by a combination of a sharp, deep attack with ES2 and a
long, low rumble with EFM1. The key to the rumble is a
custom impulse response that is sample-rate converted from
44.1 to 22 kHz and loaded into Space Designer. This boom
is controlled with rhythmic swings of the remote while hold-
ing the B button, which is the first (and, overall, the most
prominent) example of using the control data in a rhythmic
fashion. Performer one’s other remote controls a seemingly
innocuous, warbly sound in ES2 which can be quickly trans-
formed into a wall of noise with the B button. Performer
two controls chime-like sounds created with ES2. Postcard
No 4 is something of a solo for the right hand of performer
one, and showcases the melodic potential of changing notes
with the directional pad. The lengthy solo sound can be
modulated by holding the A button, and demonstrates how
simple it is to gradually morph a sound over a long period
of time using control data from the gyroscopes. The other
three remotes control percussive drum-like sounds. They
are each given constant eighth-notes through the bulk of the
piece, and can control the pitch of the sounds by raising and
lowering the remotes. Postcard No 5 is the soft, sentimental
piece of the group to accompany the unexpected discovery
of love in the text. Player one and player two have identical
sounds, and periodically play a figure in unison to accentu-
ate this. The sounds are short three-note arpeggios, and the
harmonic language is very simple and pretty. Postcard No 6
is a mix of bell-like, physical modelling sounds made with
EVD6 and played by player one and short, intense chords
made with ES2 and played by player two. The piece utilizes
both precise, unified rhythms and cadenza-like expressive
musical gestures. Postcard No 7 is a return of ideas set forth
in Postcard No 1, with a rousing pop-style ending complete
with 80s-style synth and drum machine sounds.
From a compositional standpoint, the most successful
music for Wii remotes should exploit those two most valu-
able skills that I mentioned previously: rhythmic precision
mixed with creative mapping of control data. Postcard No 1,
for example, could be executed on a synthesizer or drum
pads much more easily than with Wii remotes (of course,
practicing it helped our rhythmic control of the remotes con-
siderably). However, certain moments were extremely con-
vincing arguments in favor of the remotes: player one con-
trolling a huge boom by rhythmically swinging one arm
and then triggering a note and quickly distorting it with the
other arm in Postcard No 3; both players easily controlling
pitch shifters and delays mapped to the other player’s re-
motes in Postcard No 5; and two players producing a sweep-
ing melody and several drum sounds that smoothly rise and
fall in pitch in Postcard No 4. There are few other instru-
ments where one hand of one player can so simply and ef-
ficiently produce notes, timbral changes and engage effects.
Controlling sounds is quite straightforward and intuitive as
well; we felt that while expert control of the Wii remotes
is an important long-term goal, being able to immediately
create and perform interesting music with these new instru-
ments could dramatically improve adoption rates by other
performers and composers.
Figure 7. Example of Wii remote notation for two remotes, one performer
3.3. Notation
It was important, keeping that in mind, to devise a notational
system that was logical and easy to learn. The system we
established is based on percussion notation and contempo-
rary graphical scores, as illustrated in figure 7. For two Wii
remotes, we use something like a piano grand staff with the
left hand on the lower staff and right hand on the higher staff.
Each staff uses two lines; the two strokes sit above and be-
low the bottom line (stems always down to avoid collisions
with control data lines) and the top line is used as a reference
point for control data. Control data from the accelerome-
ters is represented with a dashed line (which better conveys
movement), while control data from the gyroscopes is rep-
resented with a solid line. A problem we encountered was
how to distinguish pitch from roll if we were changing both
simultaneously; early efforts had two solid lines which were
labeled as pitch or roll, but this was confusing and difficult
to read on sight. Fabrice Mandola, a percussion professor at
McGill University, suggested a solution that we eventually
implemented. Pitch is represented on the Y axis as before,
but rolling to the left or right causes the line to split into
two. Rolling farther in either direction widens the distance
between the two lines, and bringing the remote back to zero
causes the two lines to merge again. Left and right are distin-
guished by filling in the space between the two lines; rolling
right is represented by filling the space in with grey, while
for rolling left the space is left white or empty. Dynamics
are nowhere to be found in Postcards. This is not some-
thing intended to be absent from Wii remote notation; rather
it is a product of structuring the piece such that dynamics
were automatically generated based on combinations of the
sounds, control data, playing styles and different midi notes.
In the future, dynamics could be written in a normative way
and performers could interpret this by simply raising the re-
motes to play loud notes and lowering them to play quiet
notes.
4. PERFORMANCE
I will set out to address the major issues concerning our
wiimote instruments within the domain of musical perfor-
mance. We treat the controllers as true musical instruments
which posses expressive and artistic potential, leading to a
high level of performance, which strives to go beyond the
novelty of using video game controllers for musical applica-
tions. While similar applications for wiimotes exist, we still
strive to tackle such issues as pedagogy, effective practice,
effective rehearsal, and expressive performance. Further-
more, in the case of our performances of Postcards, having
the added element of the SoundCatcher adds to our ensem-
ble considerations and performance aesthetic.
We have to ask ourselves then, what do we gain from
using wiimotes, and why use them in the first place? Practi-
cally speaking, they are portable, and like most controllers,
they can theoretically produce any sound; in other words
they are versatile. They exist with a pre-established set of
software and are commercially available, which makes them
reliable. Like percussion, they lend themselves to a strong
visual performance where the body is involved and the audi-
ence can see a strong correlation between gesture and sound.
4.1. Extension of Percussion
As a percussionist, I will draw on the practices of my mu-
sic making as a point of comparison for the performance
issues relating to our work with wiimotes. Percussion and
wiimotes as instruments present a logical connection as their
calisthenic worlds are quite similar. Just like a percussion-
ist, a wiimote musician is faced with issues of hand inde-
pendence, hand synchronousness, wrist and finger dexter-
ity, hand positioning, etc. Fundamentally, we are making
striking motions, as does a percussionist, but the key dif-
ference with wiimotes is the lack of physical object being
struck. We have freedom in space (partially thanks to our
wiimotes not being concerned with absolute position) and
no haptic feedback, which are the biggest departures from
playing percussion. This freedom opens up a wealth of ex-
pressive possibilities.
By nature, similarities also emerge in practice techniques,
notation, and performance. Reading Patrick’s scores is in-
tuitive once one understands the various hand positions and
button usages and how they are represented on the page. In
many ways, the scores are similar to notation for a multiple
percussion setup where as many lines as instruments (or in
this case hands and hand positions) are used. In any case,
percussionists are faced with constantly changing combi-
nations of instruments and therefore different notations to
represent these setups. Using wiimotes as instruments and
learning a new notation is not a reach for percussionists who
deal with these issues in their daily musical lives.
4.2. Technique
4.2.1. Current Gestures / Striking and Buttons
Making use of what the wiimotes have to offer in terms of
measuring acceleration in specific directions, we have de-
veloped a small number of gestures which can be combined
to create more possibilities. In our case, each of the two
players uses one wiimote in each hand. The two princi-
ple gestures are the downward strike (which is sensed by
the pitch’s angular rate), shown in figure 10, and the side
strike (sensed by the yaw’s angular rate). The use of the di-
rectional buttons can be combined with either of these two
strike (figure 8 shows the player’s hand position in relation
to the buttons). In all of our work, the directional buttons
act as a pitch changing device. In this way, we can have
up to eight pitches in each hand, that is two striking posi-
tions multiplied by the four-pronged directional button. The
use of two more buttons, A and B, act as timbral control
through sweeping gestures rather than striking ones. When
B is held, the wiimote measures acceleration in two oppo-
site directions, which allows for pulsing sweeps of timbre
change. A can act as a hold on timbre change or as a way
to control modulators via the wiimotes rotation position. A
hold can be placed on the sound at any given point in the
timbre spectrum (B being held to sweep through the tim-
bre space) depending on the wiimote’s acceleration at that
moment. In most of our Postcards, we try to activate the
hold when the acceleration rate is at its highest, giving us
the most audible timbre difference for a single sound within
our sound choices. A and B are principally used for longer
sounds that merit timbre change.
Combining gestures results in more advanced technique
and more complex sonic results. For example, in Postcard
No 4 Korean Food, the left hand produces staccato sounds
using both striking positions while the right hand produces
longer sounds which are then manipulated by the A but-
ton through a ring modulator. Postcard No 3 The City also
presents advanced combinations of A / B button control and
staccato striking.
4.2.2. Gesture Evolution
Over the past few months, our gestures have evolved to what
we find most intuitive and economical. The strongest exam-
ple of the evolution of a gesture is the side strike. Origi-
nally, we turned the wiimote ninety degrees inward, through
the rotation of the wrist, so the striking motion remained
the same as the downward strike. The top of figure 9 illus-
trates this. Furthermore, I found a larger outward sweep was
needed to return to the downward striking position in order
to avoid mistriggering notes. We found a more economic
way of switching rapidly between the downward and side
strikes by triggering the side strike as an upward and inward
motion as the bottom of figure9 shows. In this sense, the
playing position for the performer changed, but in relation
to the wiimote itself, the side strike was still the same. In
other words, we were now able to create a small tick-tock
motion that avoided the outward sweep. This new gesture
combination is used extensively in Postcard No 1 Tourism.
However, this is not to say we never use the original side
strike position. For playing slower material, or when big-
ger physical gestures are desired, we can use the downward
side strike, as seen in the right hand of Postcard No 4 Korean
Food.
4.3. Practice / Pedagogy
4.3.1. How to Practice
We treat the wiimotes like any other acoustic instrument
where repetition of gestures leads to refinement and preci-
sion and eventually a vocabulary of idiomatic gestures are
developed. Because we are facing the wiimotes as a music
making device for the first time, we have the unique op-
Figure 8. Above: A button and 4 directional buttons trig-
gered by thumb; below: B button triggered by index finger
Figure 9. Above: the original playing position for the side
strike; below: the new playing position for the side strike
portunity to be practicing and discovering simultaneously.
Each of our Postcards presents different challenges that are
worked on during practice. For example, No 1 Tourism re-
quires practice of the tick-tock motion, while Postcard No 5
Love requires practice of directional button use. In a sense,
the Postcards are wiimote etudes, each having a small scope
and addressing only a few parameters of playing.
4.3.2. Pedagogy
Like any emerging digital instrument, a pedagogy must be
set up in order to give future players a framework for learn-
ing. In the case of our wiimotes, short pieces like the Post-
cards offer the kind of focus on specific techniques one would
need to become comfortable with the instrument. Because
there is no preexisting pedagogy, what I propose is merely
hypothetical at this point.
To determine a pedagogy, we must first look at the fun-
damental playing techniques and modes of expression which
can be organized into a learning hierarchy. Striking accu-
racy and directional button accuracy would be the first place
to start. These aspects of playing are the most fundamen-
tal and while intuitive, they still deserve attention as mis-
triggering can occur. Etudes using the tick-tock motion and
different pitches attached to the directional buttons, such as
Postcard No 1 Tourism, would work well to achieve this mas-
tery. Next, one would want to work with the A and B but-
tons to gain a control of timbre sweeps, modulators, holding
states, and other effects. Some time is needed to grasp the
sensitivity and range associated with these buttons. On a
more advanced level, a player would work with combina-
tions of all the controls. For example, playing melodic os-
tinatos using directional buttons and both striking positions
or using B in one hand while maintaining a striking pattern
with the other.
Beyond wiimote specific coordination, much of the other
technique one needs falls within the realm of playing per-
cussion. Issues of hand independence, hand synchronous-
ness, and hand speed for example can be developed in the
same way a percussionist would start learning snare drum or
drum set. Even before a player reaches pieces like the Post-
cards, they would want to work on Exercises such as those
found in books like Stick Control, which emphasize basic
coordination.
4.4. Rehearsal
Rehearsal for us was, in the early stages, a grounds for re-
finement of technique and discovery of more economical
ways of playing. Later, as the SoundCatcher was intro-
duced, we began finding ways to integrate it into our piece
while serving as a means of cohesion between the Postcards
and a vehicle for the narrative. Because both the Sound-
Catcher and the wiimotes had preexisting practices and id-
ioms (the SoundCatcher being previously played by Gabriel,
and the wiimotes existing as a commercially available con-
troller), we were able to move into an artistic domain where
such issues as malfunctions and controller development were
not as much of an issue. A chamber ensemble dynamic was
achieved through musical communication.
4.5. Problems / Improvements
While we are content with our Postcards and their perfor-
mance, we recognize some areas deserving of attention. One
elementary aspect of playing is the execution of simple rhythms
in time, which is more difficult than it may seem with wi-
imotes. Improvement in this area can come from hours of
practice, but also from the careful calibration of the striking
sensitivities for specific sounds. Ultimately, each Postcard
would have its own sensitivity calibration. We did a bit of
work with these calibrations, but perhaps more could have
been done. Related is the problem of mis-triggering notes.
If the correct positioning and rotation is not used, a note
may not sound, or an accidental note can sound. Improve-
ments here come from practice where the player learns the
minute details of striking boundaries. Also, depending on
the sounds being used, the players perceptions of how to
play can change. Sometimes the player must go against ten-
dencies to play big sounds big, for example. In this respect,
the player must be careful not to add superfluous gestures
that result in mis-triggering. Surprisingly small motions are
needed to trigger sounds, however this is not to say a player
can not be expressive.
An additional consideration that we did not make much
use of was dynamic control. While much of the character
of the Postcards came from their hermetic sound worlds and
musical idioms, dynamic control never really entered into
our expressivity. We had dynamics only in the sense of de-
cay and the timbral change of a decay, but no real volume
pedal. Not that such a control is necessary, but it would
surely add another layer.
Future work would also focus on building a high skill
level where what we consider advanced techniques now would
be standard practice. Given our time frame, we are still rel-
atively new to the controllers, but like with any instrument,
Figure 10. The downward strike playing position
mastery takes time.
5. CONCLUSION
We concluded our work with Wii remotes fully convinced
of their potential as Digital Musical Instruments. An in-
strument capable of rhythmic precision and dexterity that
can quickly switch into modes allowing for timbral con-
trol and that can be mapped to an effectively infinite palette
of sounds should be considered an extremely valuable tool
for composers and performers alike. Since our piece fo-
cused on scratching the surface of as many different sets of
sounds and techniques as possible, it would be ideal if fu-
ture compositions more deeply explored specific techniques
and sound synthesis possibilities. Additionally, the possibil-
ities for matching Wii remotes with acoustic instruments are
similarly endless, and future work should place Wii remotes
in chamber settings as part of an ensemble including dif-
ferent arrangements of instruments. We established a basic
foundation for Wii remote performance technique, but we
consider it to be a preliminary effort. Future work could fo-
cus on more developed pedagogical methods as well as fur-
ther explorations of different performance gestures. From a
technological standpoint, it is a concern that the Wii remote
does not provide haptic feedback; although we felt satisfied
that expressive performance is attainable, future work could
explore different types of feedback for performers.
Although using a pre-made device that was not designed
for musical applications was occasionally limiting, the ad-
vantages of having a reliable, expressive device for a very
small price outweigh the relatively minor limitations, and
we were able to compensate for many problems by devel-
oping our own software solutions and mapping strategies.
We are still in the early stages of developing composition
and performance technique for Wii remotes, and future work
should offer significant refinements.