Filipe Carlos de Albuquerque Calegario · combinação, adaptamos um método de design tradicional,...
Transcript of Filipe Carlos de Albuquerque Calegario · combinação, adaptamos um método de design tradicional,...
Pós-Graduação em Ciência da Computação
Filipe Carlos de Albuquerque Calegario
“METHOD AND TOOLKIT FOR DESIGNING DIGITAL MUSICAL
INSTRUMENTS: GENERATING IDEAS AND PROTOTYPES”
Universidade Federal de Pernambuco
[email protected] www.cin.ufpe.br/~posgraduacao
RECIFE 2017
1
Filipe Carlos de Albuquerque Calegario
METHOD AND TOOLKIT FOR DESIGNING DIGITAL MUSICAL
INSTRUMENTS: GENERATING IDEAS AND PROTOTYPES
Advisor: Geber Lisboa Ramalho Co-Advisor: Marcelo Mortensen Wanderley
RECIFE
2017
A THESIS PRESENTED TO CENTRO DE
INFORMÁTICA DA UNIVERSIDADE FEDERAL DE
PERNAMBUCO IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS FOR THE DEGREE OF DOCTOR OF
PHILOSOPHY IN COMPUTER SCIENCE.
Catalogação na fonte
Bibliotecária Monick Raquel Silvestre da S. Portes, CRB4-1217
C148m Calegario, Filipe Carlos de Albuquerque
Method and toolkit for designing digital musical instruments: generating ideas and prototypes / Filipe Carlos de Albuquerque Calegario. – 2017.
160 f.: il., fig. Orientador: Geber Lisboa Ramalho. Tese (Doutorado) – Universidade Federal de Pernambuco. CIn, Ciência da
Computação, Recife, 2017. Inclui referências e apêndice.
1. Inteligência artificial. 2. Computação musical. I. Ramalho, Geber Lisboa (orientador). II. Título. 006.3 CDD (23. ed.) UFPE- MEI 2017-213
3
Filipe Carlos de Albuquerque Calegario
Method and Toolkit for Designing Digital Musical Instruments: Generating Ideas and Prototypes
Tese de Doutorado apresentada ao Programa de
Pós-Graduação em Ciência da Computação da Universidade Federal de Pernambuco, como requisito parcial para a obtenção do título de Doutora em Ciência da Computação
Aprovado em: 31/03/2017. __________________________________________________ Orientador: Prof. Dr. Geber Lisboa Ramalho
BANCA EXAMINADORA
________________________________________________ Profa. Dra. Patricia Cabral de Azevedo Restelli Tedesco
Centro de Informática / UFPE
_______________________________________________ Profa. Dra. Veronica Teichrieb Centro de Informática / UFPE
_________________________________________________
Prof. Dr. Jônatas Manzolli Departamento de Música / UNICAMP
_______________________________________________________
Prof. Dr. Andre Menezes Marques das Neves Departamento de Design / UFPE
_______________________________________________________
Prof. Dr. João Paulo Cerquinho Cajueiro Departamento de Engenharia Mecânica / UFPE
_______________________________________________________
Prof. Dr. François Pachet Computer Science Laboratory Paris / SONY
4
To Alissa =)
5
Acknowledgement I am immensely grateful for the great support and vibrant energy that the following people
gave me in the course of these four years of research. Sou imensamente grato ao
enorme apoio e a vibrante energia que as seguintes pessoas me deram em algum
momento ou no decorrer destes quatro anos de pesquisa.
Alissa Seixas, Francisca Albuquerque, Carlos Calegario, Rodrigo Calegario, Leonardo
Menezes, Lêda Carlos, Rosa Carlos, Ana Maria Gusmão, Maribenete Menezes,
Generina dos Santos, Marcelo Gusmão, Rosângela Seixas, João Tragtenberg, Geber
Ramalho, Giordano Cabral, Marcelo Wanderley, Stéphane Huot, Johnty Wang, Ian
Hattwick, Ivan Franco, Mailis Rodrigues, John Sullivan, Lígia Teixeira, Baptiste
Caramiaux, Carolina Medeiros, Gabriel Vigliensoni, Fernando Iazzetta, Darryl Cameron,
Jerônimo Barbosa, Vânia Pontes, Celio Eyng, Thais Fernandes, Helder Vasconcelos,
Rodrigo Medeiros, Eduardo Santos, Sofia Galvão, Ricardo Brazileiro, Jarbas Jácome,
Ricardo Ruiz, Simone Jubert, Tarciana Andrade, Hermano Ramos, Clara Arruda, Clara
Vasconcelos, Jeffeson Mandu, Carlos Montenegro, Miguel Mendes, Tomás Brandão,
William Paiva, Missionário José, Helder Aragão, Yuri Bruscky, Fernando Almeida,
Renato Barros, João Marcelo Ferraz, Leo Domingues, André Araújo, Sérgio Godoy,
Sofia Freire, Pedro Luiz, Ana Cecília Barbosa, Gabriella Martins, Madyana Torres,
Emiliano Abad, Luis Arthur Vasconcelos, Jaime Alheiros, Zaca Arruda e Preta Félix.
Thank you very much! Meu muitíssimo obrigado!
6
source: https://www.reddit.com/r/funny/comments/13vd1u/the_only_diagram_i_give_a_damn_about/
“When you step into an intersection of fields,
disciplines, or cultures, you can combine existing concepts into
a large number of extraordinary new ideas”
The Medici Effect Frans Johansson
7
Abstract Last decade witnessed a considerable rise in physical, programmable, interactive artifacts.
Sensors, devices, platforms and frameworks have become more accessible and more people are
programming the physical world beyond the screen. Interactive devices for artistic expression
present challenges that are worth investigating because the interaction often needs a high level
of skill that is hard to be obtained. Therefore, interactive artistic approaches can teach valuable
lessons applicable to other levels of interaction design and human-computer interaction. One
class of artistic, physical interactive objects is the digital musical instrument (DMI). DMIs are
artifacts in which gestural control and sound production are physically decoupled but digitally
mapped. It provides freedom for a DMI designer, since several combinations are possible, but
increases the complexity of the design space. Besides, structured methods and guidelines that
would help the design have not yet been established. To address this issue, prototyping seems
to be a promising approach as they are not only a tool for testing and communicating ideas, but
also for generating them. As a DMI is a means to produce music, its prototype should provide
real-time sound feedback for control gestures. For that reason, in DMI context, non-functional
prototypes are not entirely suitable. On the other hand, the development of functional prototypes
demands more time and effort, and consequently, can be a bottleneck of iterative design. How to
provide structured and exploratory paths to generate DMI ideas? How to decrease time and effort
of building functional DMI prototypes? To deal with those questions, we propose the concept of
instrumental inheritance, that is the application of gestural and/or structural components of
existing instruments to generate ideas of new instruments. As support for analysis and
combination, we leverage a traditional design method, the morphological chart, in which existing
artifacts are split into parts, presented in a visual form and then recombined to produce new ideas.
Finally, integrating the concept and the method in a concrete object, we developed a physical
prototyping toolkit for building functional DMI prototypes: Probatio, a modular system of blocks
and supports to prototype instruments based on certain ways of holding and gestural controls for
musical interaction. The evaluation of the toolkit showed that it contributed to reducing the time
to achieve a functional prototype, and also influencing the increase in the number of cycles of
idea exploration. Besides, the users reported more musical engagement with Probatio in
comparison to a generic sensor toolkit.
Keywords: New interfaces for musical expression. Digital Musical Instruments. Idea Generation.
Ideation. Prototyping. Prototyping Toolkit.
8
Resumo A década passada testemunhou um aumento considerável em artefatos físicos,
programáveis e interativos. Sensores, dispositivos, plataformas e estruturas tornaram-se mais
acessíveis e mais pessoas estão programando o mundo físico além da tela. Dispositivos
interativos para expressão artística apresentam desafios que valem a pena investigar porque a
interação geralmente precisa de um alto nível de habilidade difícil de obter. Portanto, as
abordagens artísticas interativas podem ensinar lições valiosas aplicáveis a outros níveis de
interação e interação humano-computador. Uma classe de objetos interativos artísticos e físicos
é o instrumento musical digital (DMI), artefatos em que controle gestual e produção de som são
fisicamente desacoplados, mas digitalmente mapeados. Este desacoplamento proporciona mais
liberdade para um designer de DMI, uma vez que são possíveis várias combinações, mas
aumenta a complexidade do espaço de design. Além disso, métodos estruturados e diretrizes
que ajudariam o projeto ainda não foram estabelecidos. Para abordar esta questão, a
prototipação parece ser uma abordagem promissora, pois não serve apenas como forma de
testar e comunicar ideias, mas também para gerá-las. Como um DMI é um meio para produzir
música, seu protótipo deve fornecer, a partir de gestos de controle, feedback de som em tempo
real. Por essa razão, no contexto DMI, protótipos não funcionais não são inteiramente
adequados. Por outro lado, o desenvolvimento de protótipos funcionais exige mais tempo e
esforço e, consequentemente, pode ser um gargalo no design iterativo. Como fornecer caminhos
estruturados e exploratórios para gerar ideias DMI? Como diminuir o tempo e o esforço de
construir protótipos DMI funcionais? Para lidar com essas questões, propomos o conceito de
herança instrumental, que é a aplicação de componentes gestuais e/ou estruturais de
instrumentos existentes para gerar ideias de novos instrumentos. Como suporte para análise e
combinação, adaptamos um método de design tradicional, a caixa morfológica, em que os
artefatos existentes são divididos em partes, apresentados de forma visual e depois
recombinados para produzir novas ideias. Finalmente, integrando o conceito e o método em um
objeto concreto, desenvolvemos um toolkit de prototipação física para a construção de protótipos
funcionais de DMI: o Probatio, um sistema modular de blocos e suportes para protótipos de
instrumentos baseados em certas maneiras de segurar e controles gestuais para a interação
musical. A avaliação mostrou que o toolkit contribuiu para reduzir o tempo para conseguir um
protótipo funcional e também influenciou o aumento no número de ciclos de exploração de ideias.
Além disso, os usuários relataram mais envolvimento musical com a Probatio em comparação
com um toolkit de sensores genéricos.
Palavras-chave: Novas Interfaces para Expressão Musical. Instrumento Musical Digital. Geração
de Ideias. Ideação. Prototipação. Ferramentas de Prototipação.
9
Table of Contents 1. INTRODUCTION ...................................................................................... 121.1. Context ......................................................................................................... 121.2. Objectives .................................................................................................... 131.3. Approach ...................................................................................................... 141.4. Document Outline ........................................................................................ 14
2. CHALLENGES IN DESIGNING DMIS ........................................................... 162.1. Digital Musical Instruments ........................................................................ 162.1.1. DMI Classification .......................................................................................... 182.2. The Challenge of Multiple Combinations .................................................. 182.2.1. Gestural Controller ........................................................................................ 192.2.2. Sound Output ................................................................................................ 202.2.3. Mapping ......................................................................................................... 212.2.4. Feedback ....................................................................................................... 222.2.5. Summary ....................................................................................................... 232.3. The Challenge of Expressivity and Virtuosity .......................................... 232.4. The Challenge of Evaluation and Evolution .............................................. 242.5. The Challenge of No Previous Knowledge ............................................... 252.6. The Challenge of Multiple Stakeholders and Contexts of Use ................ 252.7. Final Considerations ................................................................................... 26
3. DESIGN PROCESS ................................................................................... 283.1. Idea Exploration ........................................................................................... 303.2. Prototyping ................................................................................................... 333.3. Final Considerations .................................................................................... 37
4. STATE OF THE ART .................................................................................. 394.1. Frameworks and Approaches for DMI Design ............................................ 394.2. Functional Prototype in DMI Design ............................................................ 454.2.1. Tools for Physical and Functional Prototyping ................................................ 454.2.2. The Trade-off Area .......................................................................................... 494.3. Final Considerations ..................................................................................... 50
5. EARLY EXPLORATION ............................................................................... 515.1. Methodological Approach ............................................................................ 515.2. Project Batebit ............................................................................................... 525.2.1. Interviews ........................................................................................................ 52
10
5.2.2. Workshops ...................................................................................................... 535.2.3. Pandivá ........................................................................................................... 545.2.4. Sandbox Wow ................................................................................................. 565.3. Summary ........................................................................................................ 58
6. PROPOSITION .......................................................................................... 596.1. Scope and Basis ............................................................................................ 596.2. Instrumental Inheritance ............................................................................... 606.2.1. Related Concepts ............................................................................................ 606.2.2. Possible Evidences ......................................................................................... 626.2.3. Discussion ....................................................................................................... 646.3. Morphological Chart for DMI Idea Generation ............................................ 656.3.1. What is Morphological Analysis? ..................................................................... 656.3.2. Morphological Chart Based on Instrumental Inheritance ................................ 676.4. Development of the Functional Prototyping Toolkit for DMI ..................... 706.4.1. Guidelines ....................................................................................................... 716.4.2. Implementation Decisions ............................................................................... 726.4.3. Physical Structure ........................................................................................... 726.4.4. Connection Slots ............................................................................................. 756.4.5. Blocks .............................................................................................................. 776.5. Final Considerations ..................................................................................... 80
7. EVALUATION OF PROBATIO 0.1 .................................................................. 817.1. Description ...................................................................................................... 817.2. Evaluation ....................................................................................................... 82
8. EVALUATION OF PROBATIO 0.2 .................................................................. 888.1. Evolution from Probatio 0.1 ........................................................................... 888.1.1. Number of Blocks and Multiple Sensors .......................................................... 898.1.2. Changing Mapping Strategy ............................................................................. 898.1.3. Sound Output Module ...................................................................................... 908.1.4. Curved Shapes ................................................................................................. 908.1.5. Connection Arm Support .................................................................................. 908.1.6. Protection and Connections to the Hub ............................................................ 908.1.7. Friction of Blocks and Slots .............................................................................. 918.2. Experiment ...................................................................................................... 918.2.1. Objectives ......................................................................................................... 918.2.2. Design .............................................................................................................. 928.2.3. Methods for Data Collection ............................................................................. 93
11
8.2.4. Methods for Quantitative Analysis .................................................................... 958.2.5. Methods for Qualitative Analysis ...................................................................... 978.2.6. Setup ................................................................................................................ 978.2.7. Technical Test Pilot ........................................................................................ 1038.2.8. Participants ..................................................................................................... 1048.2.9. Experiment Protocol ....................................................................................... 1068.3. Results .......................................................................................................... 1088.3.1. Quantitative Analysis ...................................................................................... 1088.3.2. Qualitative Analysis ........................................................................................ 1228.4. Discussion .................................................................................................... 1338.4.1. About Probatio ................................................................................................ 1338.4.2. About GSToolkit ............................................................................................. 1348.4.3. Summary of Bugs and Errors ......................................................................... 1358.4.4. Different Engagements ................................................................................... 1368.4.5. Three Profiles ................................................................................................. 1368.4.6. Limitations ...................................................................................................... 1378.4.7. Final Considerations ....................................................................................... 138
9. CONCLUSION .......................................................................................... 1409.1. Research Question Revisited ...................................................................... 1409.2. Contributions ................................................................................................ 1409.3. Limitations .................................................................................................... 1419.4. Future Works ................................................................................................ 142
REFERENCES ......................................................................................... 144
APPENDIX A - RELATED PROJECTS ......................................................... 159
12
1. INTRODUCTION “In the grand scheme of things, there are three levels of
design: standard spec, military spec and artist spec. Most
significantly, I learned that the third, artist spec, was the
hardest (and most important). If you could nail it, then
everything else was easy.” (BUXTON, 1997)
1.1. Context Last decade witnessed a considerable expansion of physical, programmable, interactive artifacts.
Sensors, devices, platforms, and frameworks have become more accessible, fostered by the
proliferation of mobile technologies, the growth of the DIY and Maker communities, and the
spread of open source and open hardware philosophies. If the question used to be "How to make
it?”, now, it becomes “What can be made of what is available?” (SANDERS; STAPPERS, 2014).
More people are experimenting with sensors and actuators, programming the physical world
beyond the screen.
Among these new interactive devices, those devoted to artistic expression present challenges
that are worth investigating, since "the essence of the artist [...] is rooted in skill, [...] which is hard
earned" (BUXTON, 1997) and should be taken into account during the design process. Interactive
artistic approaches can teach valuable lessons applicable to other levels of interaction design and
human-computer interaction.
One class of artistic, physical interactive objects is the digital musical instrument (DMI). DMIs are
devices in which gestural control and sound production are physically decoupled but digitally
connected according to a mapping strategy (MIRANDA; WANDERLEY, 2006). Sensors translate
gestures into digital data that can then be processed and mapped to sound synthesis algorithms
or modules.
DMIs cover a variety of artifacts as these examples illustrate: The Hands (TORRE; ANDERSEN;
BALDÉ, 2016), hyperinstruments (MACHOVER, 1991), Jam-O-Drum (BLAINE; FORLINES,
2002), The Hyper-Flute (PALACIO-QUINTIN, 2003), Radio Baton (MATHEWS, 2005), AKAI EWI
(VASHLISHAN, 2011), Yamaha WX5 (MIRANDA; WANDERLEY, 2006), Reactable (JORDÀ et
al., 2007), Laser Harp, Tenorion (NISHIBORI; IWAI, 2006), Novation Launchpad, Monome,
Faderfox, QuNeo (PAINE, 2013), Karlax, and Eigenharp (PAINE, 2015) (the citations do not
necessary represent the author of the instrument, but where further information can be found).
DMI design and implementation encompass a number of issues (MEDEIROS et al., 2014). One
is that the sound-control dissociation provides more freedom for DMI designers in comparison to
builders of acoustic instruments, as there are no mechanical or physical constraints. However,
the multiple input-output combinations increase the complexity of the design space and can
sometimes lead to creative paralysis (MAGNUSSON, 2010).
13
Reportedly in literature, the DMI design process is idiosyncratic (BARBOSA et al., 2015a). The
instrument is usually conceived and developed for one performer, or the roles of performer and
designer are played by one person. The discussions cover the proposition of principles, mainly
based on personal experience, and conceptual frameworks that attempt to define the relationship
between components of DMI design space, as concepts and stakeholders. The few structured
methods and processes are commonly generic, presenting no fined-grain steps that would guide
the designer during the design phase of idea generation, for instance. Thus, normally, it is
necessary for the designer to find her own suitable inspiration and ideas.
In addition, the cycle of exploring new ideas and transforming them into prototypes is an important
part of the design process (BROWN, 2008). Prototypes help to identify flaws, redirect and adjust
decisions, improve understanding of the context, and generate new ideas (WARFEL, 2009). As
the criteria for DMI design success and for their formal evaluation are not clear (BARBOSA et al.,
2015b), the DMI design process must likely rely on multiple cycles of prototyping (DAHL, 2016).
However, because the DMI is not an end in itself (such as a table or a chair), but a means to
produce music, a DMI cannot be adequately evaluated without being played. For that reason,
conventional low-fidelity non-functional prototype tools and methods are not entirely suitable for
the DMI context. For a complete understanding, DMI prototypes should be functional, reacting to
player’s actions in real-time (DAHL, 2016). This aspect demands more time and effort during
development when compared with non-functional prototypes, and can become a potential
bottleneck for iterative design process (HUOT, 2013).
1.2. Objectives Inspired by this context, our objective is related to two questions that will address conception and
implementation of DMIs:
1. How can we provide structured and exploratory paths for generating new DMI ideas?
2. How can we reduce the time and effort needed to build functional DMI prototypes?
This project aims to provide designers with directions for conception, as well as to narrow the gap
between idea and prototype. With that, we intend to boost the cycles of idea exploration, fostering
more experimentation in less time. From the perspective of iterative design cycles, by using a
straightforward and structured set of steps, we expect that designers and users can achieve a
better convergence between (a) the user’s needs, intentions, and contexts of use and (b) the
resulting, evolving DMI.
In the long run, even understanding that it is hard to know if this kind of objective is reachable, we
hope to contribute to the acceleration of evolution cycles of musical instruments. These cycles
normally take decades until the evolving instruments are actually part of human culture, and
incorporated in artistic expression.
14
1.3. Approach Considering our first question, we propose a new concept in the context of DMI design:
instrumental inheritance, which is the set of components of physical structure or playing
techniques that an existing instrument transfers to a new instrument. We also explore a design
method for idea generation based on morphological analysis (CARD; MACKINLAY;
ROBERTSON, 1991) and morphological chart (CROSS, 2000), in which existing devices are split
into their fundamental parts and then recombined to generate new ideas. In our case, we analyzed
ways of controlling and ways of holding musical instruments.
Addressing our second question, we introduce a proof-of-concept physical and functional toolkit
for prototyping digital musical instruments named Probatio (Latin word for “test, experiment, trial”)
(see Figure 1.1). Probatio provides a modular environment where users can make functional DMI
prototypes by combining parts of existing instrument controls and supports, following the
morphological chart approach.
Figure 1.1: Example Probatio in use. For the video demonstration: https://youtu.be/Ge_aj5uMgOU
It attempts to provide designers with examples and directions to generate new ideas, as well as
to reduce the gap between an idea and its working prototype. By producing an environment that
combines exploration and implementation into a single tool, it is our intent to allow the user to
create functional prototypes in less time. We expect to benefit the dialog between designer and
performer, and enable the designer-performer to explore new ideas more easily with less effort.
The two cycles of evaluation of the toolkit showed that it contributed to reducing the time to
achieve a functional prototype, and it also influenced the increase in the number of cycles of idea
exploration. Besides, the users reported they had more musical engagement with Probatio in
comparison to a generic sensor toolkit.
1.4. Document Outline • Chapter 2: we present the concept of DMI with some examples; present a list of
challenges to help to discuss the complexity of DMI design, and concludes that cycles of
idea exploration and prototyping seem to bring positive points to DMI design process.
• Chapter 3: we work on the concept of design process, focusing on idea exploration and
prototyping.
• Chapter 4: looking through the lenses of the design process, we introduce the state of
the art on frameworks, principles, methods and prototyping tools for DMI design.
15
• Chapter 5: we show our early exploration that comprises, interviews, workshops, initial
prototypes, and the insights related to each step.
• Chapter 6: we define our methodological approach and present our threefold proposition:
(1) the concept of instrumental inheritance, (2) the method of morphological chart applied
for DMIs, and (3) the toolkit intended to be used for prototyping physical functional DMIs.
• Chapter 7: we describe the version 0.1 of the toolkit and its preliminary evaluation.
• Chapter 8: we explain the version 0.2 of the system, describe a comparative experiment,
present and discuss the results.
• Chapter 9: we conclude by discussing our contributions, limitations and future works.
16
2. CHALLENGES IN DESIGNING DMIS "As Robert Moog states [...] Musical Instrument Design is
one of the most sophisticated and specialized
technologies that we humans have developed [...] [W]hen
we speak of musical instruments today, we understand
that we are talking about precisely made and finely tuned
objects." (SYLLEROS; DE LA CUADRA; CÁDIZ, 2014)
In this chapter, we examine the very concept of DMI and discuss a list of illustrative challenges
one might face when conceiving and building these artifacts. This list is not meant to be
comprehensive or exhaustive, but it helps us to attest the complexity of the area and to conclude
why it is important to have various cycles of experimentation and implementation during the
development of a new DMI.
2.1. Digital Musical Instruments Digital Musical Instrument (DMI) is a class of artistic, tangible, interactive objects intended to
articulate sound by means of gestural control. Contrary to their acoustic counterparts which follow
physical constraints, in the DMIs the control input is decoupled from the sound output (MIRANDA;
WANDERLEY, 2006). As an intermediate layer connecting these two modules, there is the
mapping strategy (See Figure 2.1). Besides, feedback beyond sound, such as haptic, luminous
are also elements of a DMI.
Figure 2.1: Digital Musical Instrument Diagram
In an attempt to discuss the concept further, Gurevich et al. (2011) argue that not all forms of
musical interactions are instrumental, thus proposing another definition: Digital Musical
Interactions (GUREVICH; CAVAN FYANS, 2011). Despite considering the relevant discussion,
in this work, we will use the term Digital Musical Instrument as an artifact composed of parts that
can be independently analyzed, with which the user interacts to obtaining a musical result.
Another term found in the literature is New Interfaces for Musical Expression (NIME). Although
NIME implies a wider definition, as it is not unique either to “Digital” or to “Instruments,” in the
present work we will use the terms DMI and NIME interchangeably.
17
DMIs comprises a plethora of artifacts, which some examples are (Figure 2.2): Haken Continuum,
a continuous playing surface that tracks position and pressure of multiple fingers; AlphaSphere,
a set of pressure sensitive tactile pads disposed as a sphere; Reactable, a combination of a multi-
touch table and acrylic cubes that control synth elements and sequencers (JORDÀ et al., 2007).
Eigenharp Alpha, a DMI in a bassoon-like appearance that comprises a matrix of keys with three
degrees-of-freedom (up, down, right, left, pressed, released), a wind controller and a ribbon
controller which control a variety of sound synthesis parameters.
Figure 2.2: DMI examples: (a) Haken Continuum, (b) Oval, (c) Ableton Push 2, (d) Eigenharp Alpha,
(e) Karlax, (f) LinnStrument (g) AlphaSphere
However, in the area, "it is hard to find artifacts that have been widely or convincingly adopted by
musicians" (MEDEIROS et al., 2014). In fact, it is considered in literature only a few virtuosi or
professional musicians (JORDÀ; MEALLA, 2014). Although relevant to the discussion, the
virtuosic use of a DMI is only one facet of the many possible options based on intentions and
contexts of use of the user.
It is reported in literature that the design process of DMI is guided by idiosyncratic approaches
(RYAN, 1991), (WANDERLEY; ORIO, 2002), (BONGERS, 2007), (WARD; TORRE, 2014),
(BARBOSA et al., 2015a), which leads to little or no room for comparison, or evolution of these
instruments.
The advance in mobile technologies, besides the DIY, maker, and open source philosophies open
the world of sensors, devices, platforms, and frameworks for a broader audience, which enforced
the tendency of the users experimenting adequate configurations for their own need.
18
Instead of denying idiosyncrasies or finding other ways to DMI development, we assume the
position of the user in the center of the DMI design process, valuing their decisions, intentions,
and contexts of use.
2.1.1. DMI Classification
Miranda and Wanderley (2006) proposes a classification continuum for DMIs (Figure 2.3) based
on the similarity of the gestural controller compared to acoustic instruments (MIRANDA;
WANDERLEY, 2006). Along the continuum, the categories are:
• Augmented musical instruments: are acoustic instruments with sensors that expand the
gestural capabilities of the performer, but maintains the gestures repertoire references of the
traditional instrument;
• Instrument-like gestural controllers: simulators of acoustic instruments, i.e. MIDI controller
version of instruments. Comparing to acoustic instruments, they present less potential of
controlling nuances but offer an expanded set of possible sounds.
• Instrument-inspired gestural controllers: present some similarity with acoustic instruments
but do not intend to simulate them;
• Alternate gestural controllers: do not hold any resemblance with existing musical
instruments.
Figure 2.3: DMI classification adapted from (MIRANDA; WANDERLEY, 2006)
2.2. The Challenge of Multiple Combinations In this section, we attempt to show the multiple combinations between inputs, mapping strategies,
and output, which are the result of the decoupling from the gestural controller and the sound
production module. We based our analysis on the model initially proposed by Rovan et al. (1997)
and refined by Miranda and Wanderley (2006) (ROVAN et al., 1997), (MIRANDA; WANDERLEY,
2006). Our objective here is to understand the DMI design space from the point of view of DMI
parts.
19
2.2.1. Gestural Controller
To better comprehend the possibilities for the gestural controller, it is necessary to understand
some characteristics of the input gestures.
Cadoz and Wanderley (2000) defines the gestural channel as a human communication means
similar to the verbal communication (CADOZ; WANDERLEY, 2000). The author defines three
functions of this channel:
• Ergotic function: (from ergos, “physical work, energy” (LUCIANI, 2007)) related to an action
that modifies and transform the environment with energy transfer between the actor and the
environment (MUSER, 2015).
• Semiotic function: delivery of meaningful information to the environment, such as thumbs
up to tell you something is correct.
• Epistemic function: exploratory movements to acquire haptic or tactile information from a
particular artifact. It can be considered as a preliminary phase to the ergotic gestures, in which
the user is experiencing the space to understand it better.
Based on the presence or absence of physical contact, Jensenius et al. (2010a) present two
definitions: manipulative gestures, which are related to physical contact, and empty-handed
gestures, a synonym of free gesture, naked gestures, or semaphoric (JENSENIUS et al., 2010).
Instrumental Gestures, or gestures related to manipulating objects, have the three functions of
the gestural channel mentioned above. These gestures can be divided into three broad
categories:
• Selection: is the choice of a particular element of the instrument, and it is not related to
adding energy to produce a sound, e.g. the position of the fingers on a flute, or on the neck
of a violin or a guitar;
o Sequential: one selection is performed at a given time
o Parallel: multiple selections are carried out at once
• Excitation: energy is added to the system. For example, the movement of the bow of a violin,
blowing a flute, pressing the key of a piano, striking a percussion instrument;
o Instantaneous: the energy is provided in a single event. It can be further divided into
percussive, when the sound event happens as soon as the physical contact begins,
and picking when the sound event happens after the physical contact ends
(ACEITUNO, 2015).
o Continuous: the energy is provided during the sound event
• Modification: the form given to some control parameter. For example, when a vibrato is
made, which is the modulation of the height of a note or the tremolo, which is the modulation
of the amplitude or volume of this note.
o Parametric: if the controlled parameter varies during the event (e.g. vibrato, tremolo)
o Structural: if the there is a qualitative change in the structure of the instrument (e.g.
the mute on a brass instrument, the sustain pedal of the piano)
20
Focusing on possible human outputs to musical instruments, Bongers (2000) lists (BONGERS,
2000): (1) muscle action; (2) blowing; (3) voice; (4) biosignals: galvanic skin response,
temperature, blood pressure, heart rate, etc. Focusing on the muscle action, the author presents
two categories based on the physiology of the muscles: isometric contraction, in which there is
no change in the muscle length or the joint angle; and isotonic contraction, in which the tension
continue the same and the length of muscle changes, causing movement. Figure 2.4 presents
the classification of muscle action.
Figure 2.4: Muscle action classification. (Extracted from Bongers (2000))
Sensors are used to allow the possible human outputs to control sound parameters. These
devices convert physical energy into electricity, and then in digital values (BONGERS, 2000).
Examples of physical energy are: “kinetic (such as pressure, torque, inertia); light; sound;
temperature; smell; humidity; electricity; magnetism; electro-magnetism (radio waves)”
(BONGERS, 2000).
By using signal conditioning and processing techniques in hardware, or in software, “devoted to
adjusting, amplifying, filtering, selecting and transducing signals” (MEDEIROS; WANDERLEY,
2014), the raw data can be manipulated to achieve more stable and adequate results (MALLOCH,
2008) (STEINER, 2005).
2.2.2. Sound Output
The output module of the DMI is responsible for synthesizing the sound based on the inputs and
mapping strategy. Examples of possible synthesis methods are oscillators and wavetables,
additive synthesis, subtractive synthesis, modulation synthesis, frequency modulation synthesis,
physical modeling, granular synthesis, vocal/formant synthesis, sampling/PCM synthesis (COOK,
2002) (MIRANDA, 1998). In this work, we do not focus on the sound generation. Thus, we
approach it at a higher level of abstraction, being more concerned with defining the possibilities
of control over the sound generation than the actual method of sound synthesis.
How to decide on the elements of the gestural controller for a new DMI? How to choose the suitable sensors to translate the gestures into digital values?
21
Schloss (1990) presents a classification for musical controls based on three levels (SCHLOSS,
1990):
• Timbral level: it is the microscopic control, in which sound properties can be manipulated in
details. It is a level that demands continuous control of the synthesis parameters in different
dimensions. For example, changing parameters such as resonance on a modular synthesizer;
• Note level: it is related to triggering notes, and controlling their execution. For example, pressing
keys of a piano;
• Process level: it is the macroscopic level based on the control of a pre-defined musical event
or sequence of events. For instance, DJ sampling.
2.2.3. Mapping
Mapping strategies are the essence of DMI (ROVAN et al., 1997) (JORDÀ, 2005a). Hunt,
Wanderley and Paradis (2002) show that even if one keeps the same inputs and outputs, but
alters the mapping strategy, the performer perceives both configurations as two different instruments (HUNT; WANDERLEY; PARADIS, 2003).
Mappings, or mapping strategy, are a set of connections between the gestural control and the
sound module.
By the nature of the mapping, it can be classified as (WANDERLEY, 2006):
• Implicit mapping: connections between input and output modules are defined by a process,
for example, machine learning, neural networks. It is considered a "black box approach"
(NORT; WANDERLEY, 2006). Examples of implicit mapping are Wekinator ((FIEBRINK;
TRUEMAN; COOK, 2009)), based on machine learning, and (CONT; CODUYS; HENRY,
2004), based on neural networks.
• Explicit mapping: the user explicitly defines the relationship between input and output by
analytically, or graphically, connecting two sets of variables (NORT; WANDERLEY, 2006).
Examples of explicit mapping systems are libmapper (MALLOCH; SINCLAIR; WANDERLEY,
2014), OSCulator (OSCULATOR, [s.d.]), and junXion v5 (STEIM, [s.d.]).
In addition to these systems, there are projects such as LoM (NORT; WANDERLEY, 2006), and
MnM (BEVILACQUA; MÜLLER; SCHNELL, 2005), that, despite using an interpolation process to
help the user map input and output parameters, the final representation can be modified
analytically (MALLOCH, 2008).
The explicit mapping’s advantage is allowing the fine-grained adaptation of the mapping by clearly
presenting analytic means to do so. Although limiting detailed modifications, the implicit mapping
strength is to encapsulate technical details related to mapping for the user.
By the number of inputs and outputs, the mapping can be classified (ROVAN et al., 1997) as:
How to choose a fitting sound synthesis? How to decide a suitable level of musical control?
22
• One-to-one: relationship in which each independent control of gesture is associated with a
musical output parameter. It is the simplest of the mappings, but usually the least expressive;
• One-to-many (divergent mapping): relationship in which an input parameter simultaneously
controls more than one musical parameter. In this mapping, few controls can possibly
generate many sound outputs. This may interfere in the sense of more expressiveness, but
limit the expressive interaction by not allowing the individualized manipulation of sound
details;
• Many-to-one (convergent mapping): relationship in which more than one input parameter
is associated with only one sound parameter. In this case, certain input variables can
influence (or modulate) the values of other input variables as well. Although more complex
than the other previously presented mappings, this convergence proves to be more
expressive than the unit mappings.
Because of their simplicity, a one-to-one mapping is considered to yield superficial user
interaction with the instrument. Research reveals (HUNT; WANDERLEY; PARADIS, 2003) that
the user feels more compelled to use a challenging musical interface than a simple one.
In an attempt to facilitate the creation of more complex mappings than the one-to-one,
intermediate layers between the controls and the musical outputs can be used. These layers aim
to process raw data by adding more semantics to their behaviors, so they were termed semantic
layers (MALLOCH, 2008). Semantic layers allow simpler visualization alternatives for instrument
designer since they encapsulate complex mappings in a one-to-one approach (HUNT;
WANDERLEY; PARADIS, 2003).
By the number of abstraction layers between the inputs and outputs, the mapping can be
classified as:
• Direct or one-layer mapping: direct connection between the raw values of the sensors and
the parameters of the synthesizers;
• Multilayer mapping: connecting raw data to intermediate layers that process values and add
meaning to them.
2.2.4. Feedback
Besides the sound feedback, DMI also presents:
• Haptic feedback: related to tactile, or kinetic references, usually related to shape, textures,
surface, or generated by vibration motors;
• Visual feedback: can be visible marks, which indicate or delimited areas of the instrument;
light sources, such as light-emitting diodes (LEDs); or images and visualizations that are
generated based on gesture or the resulting sound.
How to define an appropriate mapping strategy for a new DMI?
23
The absence of haptic feedback in DMI projects is considered a weakness when comparing to
acoustic instruments (MAGNUSSON; MENDIETA, 2007a), as it affects the embodied relation
between the musician and the artifact (PAINE, 2013). For instance, “A pianist can see and locate
a specific key before playing it, can use the resistance of the key action mechanism to help know
how hard to press the key, and can use the feeling of adjacent keys to keep track of hand position”
(DOBRIAN; KOPPELMAN, 2006). The kinetic feedback and tactile feedback can unfold additional
input channels to the brain (STEINER, 2005), therefore, helping to understand better how the
system works.
Using the generator of the feedback as a criterion, they can be classified as:
• Passive: inherent in the interaction device itself, without using any actuator, such as the
weight of piano keys, the click of the mouse, or the feeling of displacement of a computer
keyboard key (WANDERLEY, 2006);
• Active: when there is an actuator device producing movement, force, light, such as a
loudspeaker, motor, LED, etc.
In addition, depending on its source, the feedback can be:
• Primary: generated directly by the gestural control;
• Secondary: generated from the sound module.
2.2.5. Summary
In contrast to acoustic instruments, which are physically constrained by their body form that
produces sound, the DMI design allows more freedom, since the sound is digitally synthesized
by an independent module of the control interface. In DMI design, "each link between the
performer and the computer has to be invented before anything can be played" (RYAN, 1991).
The decoupling input-output allows a broad range of possibilities which enlarges the design
space, and, consequently, its exploration. Paradoxically, these numerous possibilities can lead
to creative paralysis, or difficulty to find proper ways to begin ideas exploration (MAGNUSSON,
2010).
2.3. The Challenge of Expressivity and Virtuosity An important property of a musical instrument is to enable the performer to be expressive
(MEDEIROS et al., 2014), or to “effectively convey meaning or feeling” (MERRIAM-WEBSTER,
2004). For that, it is essential that the instrument allows the performer to have a subtle control
over some features of individual notes and musical phrases, such as timing, volume, timbre,
accents, and articulation (DOBRIAN; KOPPELMAN, 2006).
How to choose the suitable feedback modality for a new DMI?
In sum, how to deal with the multiple possibilities of the DMI design space exploration?
24
In fact, for musical expression, it is not only a matter of having an instrument with an excellent
control interface, but it should also provide room for the development of control intimacy. With
high levels of control intimacy, the player can embody the instrument, i.e. “there is a transparent
relationship between control and sound” (FELS, 2004).
Virtuosity is the high developed technical skill “that enables the player to master so well the subtle
controls of the instrument that he/she can perform other cognitive activities as the music
interpretation” (MEDEIROS et al., 2014). A virtuoso does not only successfully realize a highly
difficult task, but she does it with expressivity (GUREVICH, 2009).
Acquiring a level of virtuosity with an instrument demands several years of practice
(WANDERLEY; ORIO, 2002). This fact raises concerns about the adoption of a new DMI, and
what makes it attractive for new players. In fact, Wessel and Wright (2001) discuss that it is
important for the instrument to have a “low entry fee with no ceiling on virtuosity” (WESSEL;
WRIGHT, 2001), i.e. allowing newcomers to explore the instrument with immediate musical
results, and providing room for expansion of skills and expression.
2.4. The Challenge of Evaluation and Evolution Evaluation is a critical component of the design process (BUXTON, 2007) (LOWGREN;
STOLTERMAN, 2004). In DMI, it can help the evolution of an instrument and also aids the
comparison of one instrument with another. In literature, much has been discussed about
evaluation methods: from adopting techniques from HCI (WANDERLEY; ORIO, 2002), using
qualitative approaches (STOWELL et al., 2009), focusing on the performer (BARBOSA et al.,
2011), considering the audience (BARBOSA et al., 2012), or providing tools for classifying DMIs
based on dimension spaces (BIRNBAUM et al., 2005). O’Modhrain (2011) propose different
evaluation approaches depending on the stakeholder (O’MODHRAIN, 2011). Additionally,
Barbosa et al. (2015) discuss that the term “evaluation” has different understandings within the
community (BARBOSA et al., 2015b).
Evaluation in DMI design appears to be hard because there are many levels of interacting
components and layers of complexity causing the criteria of success to be not clear (DAHL, 2016).
For that, Medeiros et al. (2014) summarize some literature topics, suggesting two categories of
criteria for success (MEDEIROS et al., 2014). One category covers how effectively the artifact
matches a given context of use, e.g. ergonomics, sound quality, visual and haptic feedback
(MONTAG et al., 2011), fine-grained gesture control, embodied relationship (ESSL;
O’MODHRAIN, 2006), efficiency, learning curve (JORDÀ, 2005a). The other category covers
criteria that normally requires a long time to be assessed, such as, instrument adoption by a
How to design a DMI that allows a novice to obtain musical results and potentially become a new user? How to design a DMI that provides room for continuous evolution of player’s techniques?
25
community, lifetime (PAINE, 2009), integration with an existing genre/style or the creation of a
new genre/style (GUREVICH, 2009), and commercial success.
In fact, studies point that evolution of acoustic instruments is based on an evolutionary process
of trial-and-error (WASSERMAN A.; CULLEN, 2015). Centuries have forged the musical artifacts
to which we are used today. Will digital musical instrument follow this process?
How to evaluate DMIs given not established criteria for success? How to evaluate DMIs considering a long time? How to accelerate the DMI design process to create and learn more in less time?
2.5. The Challenge of No Previous Knowledge Moreover, there is almost no musical repertoire created to draw the attention of potential adopters
or to contribute to the technical advances of the instrument (OORE, 2005).
Understanding, adopting and learning a conventional instrument relies on an existing body of
knowledge that comprises: (1) advanced users or virtuoso players, who demonstrate the potential
of the instrument to new players and develop playing techniques that serve as reference; (2)
musical repertoire, the set of compositions for that instrument and performed using it; (3) methods
of learning the instrument, a compilation of playing techniques presented in a structured form
focusing on practice techniques that worked previously (MEDEIROS et al., 2014).
This body of knowledge is constructed, updated and consolidated over the years, turning the
instrument into a cultural object (KVIFTE, 1988). The artifact holds an aesthetic discourse or
attitude, a regional load, an event connection, and a specific genre or style association (and even
an emotional relation).
In fact, these aspects can be crucial to foster the instrument adoption, since "some started playing
after having been inspired by some music and, in particular, the sound of the instrument"
(GREEN, 2002).
2.6. The Challenge of Multiple Stakeholders and Contexts of Use
As mentioned by (WOOD, 1997), a critical ingredient for designing systems is "understanding
potential users". DMI design is a multidisciplinary area that assembles artistic and technical
creation (JORDÀ, 2001). Performer, composer, designer, audience, manufacturer, and customer
are some of the stakeholders in the DMI context (KVIFTE; JENSENIUS, 2006) (O’MODHRAIN,
2011). Therefore, we can consider that DMIs have not only a user but an ecosystem that should
be taken into account during design. According to Payne, “any implementation of a new musical
interface must, therefore, consider the ecology of this environment” (PAINE, 2013).
How to deal with the lack a body of knowledge of a new DMI?
26
For instance, designing significant and evident connections between gestural input controls and
sound is not only a matter for the musicians, but also for the audience (O’MODHRAIN, 2011).
The definition of transparency is “the psychophysiological distance, on the player and the
audience minds, between the input and output of a device mapping” (MURRAY-BROWNE et al.,
2011). Thus, if the transparency is “opaque”, it is hard for the audience to engage with the
performance (FELS; GADD; MULDER, 2002).
Considering the performer, “good musical instruments must strike the right balance between
challenge, frustration, and boredom” (JORDÀ, 2005b). Besides that, there are different contexts
of use: the performer can do a solo or play in an ensemble; can improvise, accompany others,
play a predefined score; and can play distinct songs, repertoires, and genres.
Performing simple gestures with little effort, the musician can trigger a set of notes, pre-recorded
samples, and sound effects. Paine (2013) presents two different activities that the performer
engages during the interaction with the DMI: creation, when triggering notes and altering
parameters of sound synthesis; and control when triggering process such as samples and loops
(PAINE, 2013).
This approach has a strong relation with the conceptual framework presented by Malloch et al.
(2006) that shows three abstraction levels to categorize the performance behavior (MALLOCH et
al., 2006). The skill-based level is the rapid coordination of movement to manipulate signals in a
continuous way (e.g. someone playing violin). The rule-based level is “selection and execution of
stored procedures” (e.g. drag-and-drop pieces of music). The model-based level is the highest
level of abstraction in which the behavior goes towards a conceptual goal (e.g. live coding).
Jordà (2004) describes three levels of instrument diversities: micro-diversity, or performance
nuances, measures how to performance of the same piece can differ; mid-diversity, or
performance diversity, given one instrument how diverse is two performances played with it; and
macro-diversity, or stylistic diversity, covers how flexible one instrument is in different contexts
(JORDÀ, 2004a).
Beyond the performance, there are other contexts of use such composition; musical education;
musical therapy; musical toys, musical installations etc. Each context may demand specific
requirements or properties of the instrument; it may influence the musician satisfaction, and,
consequently, the instrument refinement and its adoption (KVIFTE; JENSENIUS, 2006) (PAINE,
2013). Those multiple contexts can substantially influence the experience in use and deal with
them simultaneously during the design process can be complex.
How to consistently include the stakeholders’ views in the design process? How to design a DMI in view of different contexts of use and diversity of the artifact?
2.7. Final Considerations From the previously discussed challenges, we can conclude that the DMI design context presents
various levels of complexity: from the micro level of mapping possibilities, passing through
27
behaviors in performance, contexts of use, and stakeholders, to the macro level of cultural
aspects. Besides, the criteria for success are loose and depend on context.
In design community, much has been discussed about a class of problems called wicked, or ill-
formulated, or ill-defined problems (BUCHANAN, 1992) (CROSS, 2006). It consists of a “class of
social system problems which are ill-formulated, where the information is confusing, where there
are many clients and decision makers with conflicting values, and where the ramifications in the
whole system are thoroughly confusing” (BUCHANAN, 1992). In fact, there are already studies
that define DMI or NIME design as a wicked problem (DAHL, 2012) (DAHL, 2016).
A characteristic of these problems is that normally they do not have a definitive formulation, and
the problem definition always comes in pair with the problem-solving. In that situation, cycles of
idea exploration and prototyping play a major role in the process (JOBST; MEINEL, 2014) (VON
THIENEN; MEINEL; NICOLAI, 2014), as the cycles are not only meant to generate, test, and
communicate ideas but also to interpret and create meaning, sometimes redefining the problem
after learning in the process (FALLMAN, 2003).
In DMI context, the artifact is not an end, but a means to produce sound and music. To have a
clearer understanding of how the DMI behaves and validate if it is adequate to an intention or
context of use, it is important to have real-time audio feedback of the control gestures.
Accordingly, conventional low-fidelity non-functional prototype tools and methods employed in
HCI, such as the paper prototype or Wizard of Oz (BUXTON, 2007), are not entirely suitable for
the DMI context. For a richer comprehension, DMI prototypes should be functional, reacting to
player’s actions in real-time (DAHL, 2016).
However, the development of functional prototypes demands more time and effort, and
consequently, they can become a hindrance for iterative design (HUOT, 2013). During the design
process, more iteration leads to mature results to the context of use and intention. The more
cycles of prototyping the process has, the better the results are expected to be (BROWN, 2009).
To deal with the wicked nature of DMI design and to focus on designing DMIs for a broader
adoption, we raise two questions for guiding our research:
• Regarding the complexity of the DMI design space, how to provide structured and exploratory
paths for generating ideas of new DMIs?
• Considering the functional requirement of DMI prototypes, how to decrease the time and effort
of building them?
In the next chapter, we deepen our discussion in the design process based on design literature,
focusing on the phases of idea exploration and prototyping.
28
3. DESIGN PROCESS "Design is the creation process through which we employ
tools and language to invent artifacts and institutions. As
society has evolved, so has our ability to design." (OWEN,
1993).
The purpose of this chapter is to introduce the concept of design process and its phases, focusing
on idea exploration and prototyping. Metaphorically, we would like this chapter to be a pair of
glasses, through with we can analyze initiatives in DMI literature (discussed in the next chapter)
that address the conception and implementation of new DMIs. Diving into the metaphor, idea
exploration would be one of the lenses of our glasses and prototyping would be the other (Figure
3.1).
Figure 3.1: Understanding the design process focusing on idea exploration and prototyping
The design process can be understood as a sequence of activities, or methods, that are
performed in series, or in parallel, in order to design something (CROSS, 2000) (JONES, 1992).
Besides guiding the designer to take decisions, the design process can be used to explain the
designer’s activities to users, collaborators, and students (DUBBERLY, 2010).
Cross (1984) explains that logical analysis and creative thought are present and necessary for
the design process. By using structured and clear ways of understanding the process, these two
forms of thinking can take place without cognitively loading designer’ mind or relying on the
designer’s inner inspiration timing (CROSS, 1984).
In Figure 3.2, we present examples of processes from different areas such as engineering design,
mechanical design, creative thinking, user-centered design, and innovation. Finally, inspired by
these diverse design processes, we propose a set of phases that can be used to analyze the DMI
design context.
29
Figure 3.2: Examples of Design Processes’ Phases based on literature. (CROSS, 2000), (PAHL et al.,
2007), (DUBBERLY, 2008), (PUCCIO; CABRA, 2012), (BROWN, 2008), (IDEO, 2011), (MAURYA, 2012), (NEVES, 2014)
The proposed phases are used in this work for analysis only, and it is not a definitive or
comprehensive proposition for DMI design process. Additionally, we use the concept of design
space as a constrained set of design possibilities that leaves some dimensions open for
exploration (BEAUDOUIN-LAFON; MACKAY, 2000).
• Problem / Design Space Definition: related to understanding the concepts of musical
instruments, such as the stakeholders, the scenarios, the common knowledge of the area,
understanding the user’s intention and context of use, and defining the restrictions that define
the design space of the project.
• Idea Exploration: related to exploring possible paths in the design space, generating and
selecting ideas that conform to user’s intentions and contexts of use.
• Prototyping: related to concretizing the abstract ideas into an artifact that can be utilized,
and tested, with which the user can interact.
• Evaluation: related to validating whether the artifact is adequate to user’s intention and
context of use.
Figure 3.3 illustrates how we understand the relationship between the proposed phases of DMI
design process.
30
Figure 3.3: Proposed phases for analyzing DMI design process
In the next sections, we will focus on idea exploration and prototyping, since these phases are
emphasized in this work.
3.1. Idea Exploration “Design conceptualisation can be defined as creating an
idea, gradually maturing its meaning and eventually
expressing the understanding through representations like
words, drawings or models.” (CAPJON, 2005)
Ideation methods provide a prescription (normative
procedure) on how to overcome certain blocks to creativity
(HERNANDEZ; SHAH; SMITH, 2010)
Idea Exploration can be understood as the process of conceiving and testing ideas that can be
useful for the accomplishment of some desired result (REINIG; BRIGGS, 2008) (PUCCIO;
CABRA, 2012).
We consider that idea exploration comprises idea generation (also called “ideation” in some
references) and idea selection (also mentioned as “idea evaluation” (PUCCIO; CABRA, 2012)),
that are respectively divergent, and convergent ways of thinking about solutions based on the
design space. According to Puccio (2012), creativity results in novel and useful outcomes, and
idea generation and idea evaluation can be respectively related to the search for novelty and the
pursuit of usefulness.
It is important to make a clear differentiation between idea evaluation and the design process
phase, evaluation. The latter concerns the validation of a tangible artifact and its attainment to
the outcome intentions, and the former still on an embryonic, or abstract stage of exploration.
31
In an attempt to compare the idea generation methods, Shah et al. (2000) present a classification
based on the essence of the methods (Figure 3.4) (SHAH; KULKARNI; VARGAS-HERNANDEZ,
2000). For the authors, intuitive methods focus on fueling the unconscious thinking, and logical
methods rely on rational approach towards a problem.
Figure 3.4: Classification of idea generation methods adapted from (SHAH; KULKARNI; VARGAS-
HERNANDEZ, 2000)
Although understanding fewer details about how intuitive methods affect the designer’s mind,
these methods are related to inducing more novel results (SHAH; KULKARNI; VARGAS-
HERNANDEZ, 2000). The subdivisions proposed by the authors to classify intuitive methods are:
• Germinal: methods intended to be used when the designer has no previous solutions or
ideas, as an initial step;
• Transformational: methods that transform existing ideas to generate new ones;
• Progressive: methods based on applying repetitive steps;
• Organizational: methods that help the designer to group the already generated ideas;
• Hybrid: combination of the aforementioned methods.
For logical methods, the subdivisions are:
• History based: methods that leverage existing solutions, which are typically compiled in
catalogs or archives;
• Analytical: methods based on systematical exploration of relations, causes and effects, and
wanted or unwanted characteristics of the already generated ideas.
According to Shah et al. (2000), key components of an idea generation method are "mechanisms
that are believed to promote idea generation intrinsically or to help designers overcome specific
32
mental blocks". The authors surveyed literature in cognitive psychology and engineering design
to recognize these components from a variety of ideation methods and collect evidence of their
usefulness. The authors provide the following list of idea generation promoters:
• Combinatorial Play or Synthesis: allow the combination of parts, modules, components, or
other ideas to achieve new results.
• Use of analogies and metaphors: mapping between familiar aspects of an item into an
unfamiliar context;
• Imagery/Sketching: presence of pictorial representation during the ideation;
• Feedback: continuous feedback whether the generated ideas are leading towards the goals;
• Constraints: imposed limits that allow the designer to focus on specific aspects of the set of
possibilities;
Additionally, idea generation “tackles” are components that help to overcome mental blocks.
Some examples are mentioned by Shah et al. (2000) and Hernandez et al. (2010):
• Provocative Stimuli: Display correlated and uncorrelated materials to designers, in the form
of images, texts, sounds, objects.
• Suspend Judgment: Delay early decisions that may put ideas away;
• Flexible Representation: use means that can be easily understood changed such as
graphical representation.
• Frame of reference shifting: modify the way the goals of the project are being absorbed
visually or understood;
• Incubation: force the designer to delay some aspects of the ideation process to allow
unconscious processing to happen.
• Example exposure: related to Provocative Stimuli, but in this component, solutions for the
same problem are shown to the designer. Literature may indicate that the presence of this
component may cause design fixation, which is when the designer unconsciously focuses on
specific aspects of an artifact, neglecting others, thus negatively interfering on creative
outcome (VASCONCELOS; CRILLY, 2016).
• Random connections: allow random combinations or connections between the explored
elements;
• Emphasis on quantity: focus on the generation of a high number of ideas. However, Reining
and Briggs (2008) argues that relation between the number of ideas and the quality of ideas
are not always direct due to cognitive and solution space limitations (REINIG; BRIGGS,
2008).
• Emphasis on variety: focus on producing results that have diverse characteristics from each
other.
Figure 3.5 presents five examples of idea generation methods highlighting their key components.
33
Figure 3.5: Comparison between idea generation methods adapted from (SHAH; KULKARNI;
VARGAS-HERNANDEZ, 2000)
Smith (1998) presents the concept of active ingredients in idea generation methods. The author
analyzed 172 methods of idea generation and distilled 50 categories of prescribed actions by
which idea generation techniques affect the designer thinking (SMITH, 1998). The following mind
map (Figure 3.6) shows an abbreviated version of the ingredients.
Figure 3.6: List of Active Ingredients for Idea Generation adapted from (SMITH, 1998)
3.2. Prototyping “When something is truly novel, we cannot plan it into
existence, but we need experimentation to learn through
trial-and-error.” (PASSERA; KAERKKAEINEN; MAILA,
2012)
Prototypes are the reduced implementation of an idea focusing on some aspect such as function,
or form. Gill (2011) discusses how the term prototype is used in different areas. Industrial
34
designers tend to focus more on appearance, engineering designers normally are more
concerned with the artifacts’ functionality, and for an interaction designer or software engineer,
the prototype can mean a simulation of a user interface, for instance. (GILL; SANDERS; SHIM,
2011).
Due to these diverse areas, Gill (2011) concludes that trying to find a common understanding is
hard, but some aspects seem to be recurrent such as the objective (explorative or evaluative
prototypes), or the level of prototypes details (low-fidelity or high-fidelity) (GILL; SANDERS; SHIM,
2011). Exner (2015) proposes three categories to investigate the prototyping process (EXNER et
al., 2015): objectives (explorative, experimental, evolutionary), dimensions (form, material,
concept, principle, process, functions, requirements), and fidelity (high, low, mixed)
Describing dimensions in detail (EXNER et al., 2015):
• Form study: related to visual features of an artifact and its final look
• Material study: related to the experimenting different materials to accomplish the desired
look-and-feel
• Proof of concept: to verify feasibility of concepts (for example, the use a new kind of
technology)
• Proof of principle: to check the applicability of a principle (e.g. biological principles)
• Proof of process: in the context of service development, is related to checking the
procedure, and completeness of a service.
• Proof of function: to verify if specific functions or requirements have been satisfied.
Prototyping objectives:
• Exploratory: focus on the use of quick and straightforward prototyping methods, such as
paper prototype, in order to explore initial ideas and concepts. It is also called throw-away
prototypes because the important part is not the material, but the knowledge gained during
the process.
• Experimental: aims to verify if principles and requirements are being fulfilled. Normally,
happens in later stages in the design process.
• Evolutionary: is intended to be used as a scaffold that is constantly being modified and
enhanced. “e.g. by automatically digitalizing sketches of user-interfaces in a working
smartphone app” (EXNER et al., 2015).
Prototype fidelity is related to the level of details and similarity with the final product (EXNER et
al., 2015). Although fidelity is a term often used in literature, Beaudouin-lafon (2000c) discusses
that precision is preferable, because it focuses on the prototype itself and not on a product that is
not defined yet (BEAUDOUIN-LAFON; MACKAY, 2000). The level of fidelity or precision of
prototypes can be classified as:
• Low: simplified representation with limited details, focusing on few aspects of the idea.
Normally associated with initial stages of the design process, this level of prototyping is
35
considered to be useful for collecting usability data at a low cost (GERBER; CARROLL,
2011).
• High: a greater number of details, focusing on more than one aspects of the idea. In
engineering design or industrial design, it may be associated with the concept of not being
distinguishable from the final manufactured artifact (HORVAT, 2011).
• Mixed: combines elements of low-fidelity and high-fidelity prototypes in one approach. For
example, when sketches are used as a representation for an interactive prototype of a mobile
user interface (LIM; STOLTERMAN; TENENBERG, 2008).
Additionally, Nakamaru (2016) expands the concept of fidelity with a two-dimensions classification
considering the axes appearance and function (NAKAMARU, 2016) (Figure 3.7). With this
categorization, the author highlights two combination possibilities, appearance, and functional
prototypes, that probably were blurred in the one-dimensional fidelity classification. Further, in the
text, we decided to use this classification for comparing different prototyping tools for DMI.
Figure 3.7: Two dimension prototype classification proposed by (NAKAMARU, 2016)
Concerning the benefits of using prototypes, Angesleva (2016) enumerates eight topics (ÄNGESLEVÄ et al., 2016a):
• Help to understand complex concepts
• Allow the visualization of abstract ideas
• Enhance communication, since they remove cultural and linguistic barriers
• Exercise the focus, as they are built to experiment specific points
• Test functionalities and related them to requirements
• Build the ground on which other ideas can develop
• Refine users’ interests
• Allows a better comprehension of users’ interaction
36
According to (DOW, 2011), prototyping plays a central role in product design and interaction design, since it helps to identify flaws in the artifacts, to redirect and adjust them, to have a better
understanding of how they work and to generate new ideas (WARFEL, 2009).
As said by Gauntlett (2014), during conceptual design it is difficult to hold all detailed, and
complex information about an idea in our head at once (GAUNTLETT, 2014). Donald (2002)
introduces the concept of ‘external memory field’, which can be a note, a drawing, or a physical
model, that enhances our ability to deal with those complex details, and unburdening our minds
(DONALD, 2002).
Accordingly, Beaudouin-lafon (2000c) states the importance of prototyping by highlighting the
capability of generating “concrete representations of new ideas and clarifying specific design
directions” (BEAUDOUIN-LAFON; MACKAY, 2000). Besides, the author emphasizes the
tangibility of the prototype, stating that: “[prototype is] not an abstract description that requires
interpretation." (BEAUDOUIN-LAFON; MACKAY, 2000)”.
Additionally, prototypes play a central role in the visualization of ideas, since they transform
thoughts into tangible representations that can be easily read and shared (PASSERA;
KAERKKAEINEN; MAILA, 2012).
Much has been discussed in the literature about the benefits of producing numerous concepts
during the design process (ELSEN et al., 2012). Often, the quantity of ideas is associated with
quality of creative outcomes. However, studies suggest that there is a trade-off of quantity and
creative results. Relatedly, it seems to be well-stablished that more iteration leads to mature results to the context of use and intention (CAMBURN et al., 2015). The more cycles of
prototyping the process have, the better the results are expected to be (BROWN, 2009).
Israel et al. (2016) performed a focus group with prototyping experts from different areas in order
to discuss the future perspective of prototyping (ISRAEL; BÄHR; EXNER, 2016). Topics such
as hybrid and modular prototypes were associated with blurring the barrier between physical and
digital prototypes, leading to a deeper validation of user experience. Another highlight was the
importance of quick changes between physical and virtual prototypes, focusing on the evaluation
of the interaction between the user and the artifact.
In the dynamic contexts of interactive applications, where it is important to assess the responsive
nature of the system, the literature highlights limitations of the use of low-fidelity prototyping
approaches (LIM et al., 2013). Therefore, functional prototypes seem to be the most suitable
option to study interaction, due to the continuous cycles of action and reaction between user and
system.
However, the development of functional prototypes demands more time and effort (LIM et al.,
2013), besides requiring more technical expertise from the designer (ROECK et al., 2013).
Consequently, they can become a hindrance in the design process (HUOT, 2013).
To illustrate the time and effort to build functional prototypes, Sadler (2016a) studied an expert
during the prototyping process of a smart shoe. The prototyper used a wearable camera on his
37
chest during all the development. The results show that the expert took more than 15 hours to
produce a functional prototype, and used more than 30 different tools. Besides, there are shifts
of contexts such as dealing with structure, electronics, and programming (SADLER, 2016).
Nakamaru (2016) depicts the functional prototyping as a customer journey map divided into five
stages: Planning, Preparation, Development, Use, Keep/Destroy (NAKAMARU, 2016).
According to Sadler (2016a), that the number of technical interruptions during prototyping
negatively affects the creative process, since it generates an increase in cognitive load, and
deviate the user from the task of creative experimentation (SADLER, 2016).
“[Thomson's Rule for First-Time Telescope Makers] It is
faster to make a four-inch mirror then a six-inch mirror than
to make a six-inch mirror." (BENTLEY, 1985)
Isa et al. (2015) emphasize that literature is divided into whether prototypes should be used in the early stages of design or not. The authors who state the latter emphasizes the high cost
and time to build the prototypes, suggesting to using prototypes only when actually needed. On
the contrary, other researchers defend prototyping in early stages, because they enhance the
communication between the users, the designers, and possibly other stakeholders (ISA; LIEM;
STEINERT, 2015).
Elsen (2012) recommends that the “prototyping should begin as soon as possible during the
design process” (ELSEN et al., 2012). The author mentions that a commonly performed practice
in the preliminary stage of the design process is to use low-fidelity prototypes since they are quick
to build with associated low cost.
“Rapid and early prototyping enables learning through
making.” (SANDERS, 2013)
Furthermore, Valamanesh et al. (2013) defend that prototypes should be built in the early steps
of the design process, and can also be used as tools for idea generation, “since a physical artifact
enables designers to be exposed to unlimited perspectives and combinations” (VALAMANESH;
SHIN, 2013). Viswanathan et al.’s (2015) study confirms that prototyping physical artifacts aids
to remove incompleteness in initial ideas leading to better outcomes (VISWANATHAN et al.,
2015). Besides, Youmans et al. (2011a) emphasize the property of reducing the cognitive load in
artists and designers, as prototypes can store ideas for further development (YOUMANS, 2011).
3.3. Final Considerations In this chapter, we presented the definition of the design process, compared different descriptions
of its phases, and summarized them in four phases that we propose to use in the DMI context:
design space definition, idea exploration, prototyping, and evaluation. Based on the scope of this
work, and relating to our first question, we focused on the description of idea exploration, which
comprises the cycles of idea generation and idea evaluation, and prototyping. We highlight that
38
idea generation methods are a set of systematic steps whose components or active ingredients
foster the creative thinking and/or help to overcome mental blocks.
We discussed that prototyping is important to the design process because it does not only
concretize ideas into partial implementations that can be evaluated, but it also impacts the idea
generation, redefining initial impressions and understandings. Prototypes should be built in early
stages of the design process, as quick as possible, and with as many cycles as possible because
the number of cycles is often associated with mature outcomes. We also argued that functional
prototypes provide a better comprehension of the ideas, but they are often more difficult to build.
With the concepts discussed in this chapter, we aimed to be equipped with the necessary
understanding to analyze the DMI literature in search of initiatives that deal with the phases of
idea exploration and prototyping.
39
4. STATE OF THE ART In this chapter, firstly, we present initiatives in DMI literature such as concepts, principles,
frameworks and processes that can be related to DMI design, and specifically, we look for
structured steps for idea exploration. Then, we analyzed commercial products and DMI literature
in search for prototyping tools that can be suited for DMI prototyping phase. Following our glasses
metaphor, in sum, we will discuss initiatives and tools focusing on idea exploration and
prototyping (Figure 4.1).
Figure 4.1: Analysis of DMI literature and commercial products focusing on idea exploration and
prototyping
4.1. Frameworks and Approaches for DMI Design To help us analyze different approaches or initiatives in DMI literature, we opted to define some
categories inspired by definitions presented by O’Modhrain (2011) (O’MODHRAIN, 2011). We
use the term conceptual framework as a theoretical set that describes the relationship between
elements of a certain design context. We understand that a concept is an isolated piece of
knowledge, which can be part of a framework or be presented alone as the result of a study or
discussion. Principles, guidelines or recommendations (that can also be part of frameworks) are
attempts to transfer experience in the form of points to be achieved (often they do not show how
to achieve the point). Finally, we consider that processes and methods are a set of structured
steps towards a specific goal.
In early years, only academic laboratories had access to expensive computers and sensor
platforms that could provide the immediacy response between gestural controls and sound
production (BATTIER, 2000). With the popularization of personal computers and their increasing
processing power, the real-time interaction became available for a broader audience. This fact
seems to have influenced the growing discussion in the last years about DMIs, or NIME.
40
Jordà (2004) argues that low-level research that tries to solve only parts of DMI design is
necessary, but insufficient and integral studies that carry a holistic view of the area should be
conducted, since "very few attempts are being made" (JORDÀ, 2004b). Bongers (2007) states
that the literature in the area lacks established "guidelines and approaches for [the] complexity"
(BONGERS, 2007). Restating that this issue might not have changed in time, Jordà (2014)
mentions that "general and formal methods that go beyond specific use cases have probably not
yet emerged. Will these be the El Dorado or the Holy Grail of NIME research?" (JORDÀ; MEALLA,
2014).
"While standardisation may not be a must (maybe not
even desirable), highly idiosyncratic instruments which are
often used only by their creators may not be the best sign
or strategy for a serious evolution in this field." (JORDÀ,
2004c).
As a direct consequence of this lack of established guidelines or methods, the conception and
implementation of some instruments might not leverage the accumulated experiences and best
practices from the others.
By analyzing the literature, we could realize the effort of the community in defining frameworks,
concepts, principles, guidelines, and recommendations in DMI design. However, only a few
references present processes or methods that could guide the designer into an initial path in the
design process.
Some authors present individual contributions and reflections in the form of concepts, such as
Fels et al. (2002), who define transparency, or the ease of understanding the mappings of an
instrument, that can be achieve with the use of metaphors and it is related to instrument
expressivity (FELS; GADD; MULDER, 2002). Besides, Essl and O’Modhrain (2006) came to the
conclusion that the sensorimotor experience is an important point for taking into account when
developing engaging musical instruments (ESSL; O’MODHRAIN, 2006).
Wanderley and Orio (2002) provide a set of recommendations to formulate better ways to define
musical tasks that can be tested using established concepts and methods from HCI
(WANDERLEY; ORIO, 2002). Additionally, Blaine and Fels (2003) introduce a list of "elements of
design" related to collaborative musical interfaces (BLAINE; FELS, 2003).
Additionally, Cook (2001) and Cook (2009) present a set of principles (Figure 4.2) for designing
computer music controllers based on the author's experience with related projects (COOK, 2001)
(COOK, 2009). The principles cover topics related to artistic, and technical aspects of instruments
development. As explained before, these principles are abstract design goals, but they do not
present detailed information on how these goals should be accomplished (O’MODHRAIN, 2011).
41
Figure 4.2: Set of Principles proposed by Perry Cook. Adapted from (COOK, 2009)
Additionally, Birnbaum et al. (2005) present a graphical tool that aims to facilitate communication
and support design decisions (BIRNBAUM et al., 2005). Their proposed dimension space (Figure
4.3) consists of a radar graph with seven-axis representing comparative aspects of DMIs such as
required expertise to play, musical control, feedback modalities, degree of freedom, inter-actor,
distribution in space, and role of sound.
Figure 4.3: Dimension Space proposed by (BIRNBAUM et al., 2005)
The mentioned initiatives contribute to DMI design by attempting to explain the components of
the area and their relationship. They provide little or no initial structured path for a designer to
42
start conceiving and developing a DMI. These concepts and frameworks are more suitable for
validating ideas or classifying an already built instrument (a posteriori) than generating them (a
priori). In fact, Marquez-Borbon et al. (2011) highlight that NIME literature shows attempts to
“categorize and situate existing or newly designed musical devices in the growing body of
exemplars”, but emphasize that there only a few tries going towards generative approaches
(MARQUEZ-BORBON et al., 2011).
Miranda and Wanderley (2006) present a list of steps normally applicable for designing a new
DMI (Figure 4.4) (MIRANDA; WANDERLEY, 2006). The approach is more suited as a
recommendation for delimiting the design space than a method for idea exploration, as it gives
little guidance on how to accomplished the steps. It is important for the designer to know that is a
fundamental information to have, but she typically needs to find her proper inspiration for
generating the ideas.
Figure 4.4: Topics on designing new digital musical instruments extracted from (MIRANDA;
WANDERLEY, 2006)
Drummond (2009) studies the context of interactive music systems and formulates a set of
definition, classification, and models in an attempt to summarize different views for what the
author call a cross-disciplinary field (DRUMMOND, 2009).
O'Modhrain (2011) reflects about different evaluation methods for distinct stakeholders in the DMI
context, such as performers, audience, composers, designer, manufacturer, and customers
(O’MODHRAIN, 2011). The author guides the evaluation by proposing methods related to each
stakeholder. This is an example of a method used to validate and, logically, for validating one
expects to have already an idea. This is supported by Jordà et al. (2014), who mention that the
discussion regarding evaluation can influence instrument design (JORDÀ; MEALLA, 2014), but,
in general, the evaluation deals with a posteriori aspects, and possibly lacks elements of formative
or generative thinking.
43
Morreale et al. (2014) gather different concepts from the literature to build a unifying framework
for digital instruments and musical installations, called MINUET (MORREALE; ANGELI;
O’MODHRAIN, 2014). The authors organize the topics into two major groups: Goal and
Specifications. By describing people, contexts, activities, and technologies, the designer would
have covered the core elements in DMI conception. The framework diagram is shown in Figure
4.5. Although MINUET reduces the complexity of DMI design space, the designer has to generate
the ideas and then "plug" the ideas on the framework. For example, the framework does not
provide a set of examples of technologies, or contexts, (even if this initial description was
superficial) that the designer can choose from and start thinking about them.
Figure 4.5: MINUET framework extracted from (MORREALE; ANGELI; O’MODHRAIN, 2014)
Wallis et al. (2013) discuss the property of long-term engagement of musical instruments and
what can be learned to apply in HCI development process (WALLIS et al., 2013). The authors
present a set of heuristics meant to be used in HCI projects from idea generation throughout the
evaluation. The heuristics cover three aspects of long-term practice with musical instruments:
mastery, autonomy, and purpose. Although the authors present the heuristics as a method for
idea generation, the descriptive nature of the list makes it closer to the definition of the design
space.
44
Fyans et al. (2012) apply participatory design to engage performers and spectators into the design
process in an attempt to extract relevant information about the usage and perception of the
instruments (FYANS et al., 2012). The authors point out that since the design of DMI is an open-
ended process, the definition of a goal is generally subjective. This yields in issues when defining
the modifications within the design cycles.
Besides, (SYLLEROS; DE LA CUADRA; CÁDIZ, 2014) presents an instrument created based on
a cyclic design process centered in a group of users. The authors discuss concepts of user-
centered design, personal identities, and interactive behaviors to establish their method, and also
rely on users focus groups to discover requirements that inform the proposed solution.
Additionally, one of the design process phases takes into account the number of movements of
user's joints after a sound stimulation session. Considering the osteokinematic motions, user’s
movements are ranked as presented in Figure 4.6. Even though a structured method is
presented, and imposes some restrictions in the search space of user’s gestures, the authors do
not discuss the aspects that lead them to generate the ideas of the resulting instruments.
Figure 4.6: Osteokinematic motions and the percentage of motions performed by the users
(adapted from Sylleros et al. (2014))
In sum, there are numerous attempts to build up a body of knowledge on DMI design but few
structured processes and methods which lack paths on how to achieve the mentioned results
regarding instrument ideas. Although the discussions about defining elements of the area are
necessary to reduce the complexity of DMI design space, we believe that structured and
exploratory paths could help DMI designers to conceive instrument ideas, and also could
contribute to kick start and accelerate new DMI projects.
In the context of engineering design, Pahl et al. (2007) highlight the importance of having explicit
and structured steps during the design process (PAHL et al., 2007). The authors present that the
weaknesses of not following a structured method are: “the right idea rarely comes at the right
moment”, “the result depends strongly on individual talent and experience”, “there is a danger
that solutions will be circumscribed by preconceived ideas based on one’s special training and
experience”. That is why we find it important to provide methods for idea generation.
45
4.2. Functional Prototype in DMI Design There is a considerable amount of hardware and software tools already available that could be
used to build functional DMI prototypes. Some examples are:
• microcontrollers environments (Arduino, Raspberry Pi, Beaglebone, Teensy),
• sensor kits (Infusion Systems, littlebits, makey makey)
• MIDI controllers (keyboard, wind controllers, percussion controllers)
• general purpose programming languages (C, C++, Java)
• audio-oriented programming languages (CSound, SuperCollider, Chuck, Pure Data,
Max/MSP),
• creative programming environments (Processing, openframeworks, Cinder, Scratch)
• applications for mappings (libmapper, iCon, OSCulator, juxion, Wekinator), and
• digital audio workstations (Logic Pro, Ableton Live, Pro Tools, GarageBand, Reaktor,
Tassman).
The designer must, therefore, be a polyglot in a plethora of technologies to be able to "hack",
adapt and integrate specialized tools effectively to obtain non-trivial functional prototypes
(HARTMANN et al., 2005). Since the tools are fragmented, the shift of context between multiple
development areas (mechanical structure, electronics, programming, and sound design) and their
technical details directly affects the duration of prototyping cycles. For further analysis, we
selected a set of hardware tools that can be used to build physical functional prototyping.
4.2.1. Tools for Physical and Functional Prototyping
In this section, we analyze academic and commercial systems and devices that can be used to
help prototyping physical and functional DMIs, whose objective is to allow the experimentation of
musical interactions. The list of the projects and references can be found in Appendix A. Besides,
the detailed description of each tool is out of the scope of this analysis.
Our analysis criteria are inspired by the two-dimensional prototype classification proposed by
(NAKAMARU, 2016). However, instead of using the concept of appearance, we adopt the notion
of physical structure. For DMIs, this criterion is relevant given the intimate physical connection
between the musician and the instrument. Supporting that concept, Paine (2003) performed a
study with musicians that showed their “strong need for a physical connection with their
instrument”, and defines the physical relationship between performer and instrument as “crucial
step in the development of new musical interfaces” (PAINE, 2013).
Because of that, we decided that the scope of this analysis is restricted to systems whose
resulting prototypes’ interaction is based on manipulative gestures. Therefore, systems that deal
exclusively with empty-handed or free gestures are not considered in the present analysis.
We are interested in analyzing the features of the projects that help the designer to achieve
physical, functional prototypes. Although not being a continuous variable, didactically we
46
attempted to present the results in a continuum in both axes from characteristics that demand
more effort and time to less effort and time. According to Lidwell et al. (2010), flexibility is accepted
to be inversely proportional to usability (LIDWELL; HOLDEN; BUTLER, 2010) in a given system.
In our case, we consider that the usability of a prototyping tool is how quick and easy is to obtain
a functional prototype. For example, a textual programmed hardware microcontroller allows the
designer to build a great variety of functional prototypes. In our interpretation, we consider that
this is a case of high flexibility. However, the designer needs to dedicate a lot of time and effort
to overcome technical barriers to accomplish the tasks of programming and building the structure
of the prototype. On the other hand, a system with a fixed structure and a fixed set of interaction
can allow immediate experimentation but lacks possibilities to modifications, i.e. low flexibility.
For the physical structure axis, the points of the scale are presented in order, from more
demanding (more flexible, less immediate use) to less demanding (less flexible, more immediate
use) regarding quickness and ease to accomplish results:
• Only a board: no physical structure other than the board. In this case, all the components
that will serve as a physical interface to the user should be acquired elsewhere. The designer
has to build the physical structure, acquire the sensors or input devices, and define the layout.
• Board with sensors: the physical interface comprises the structure of raw sensors. The
designer has to build the physical structure and deal with arranging the layout of the sensors.
• Supports and modules: there are one or more physical supports where the modules can be
placed, but cannot be connected together. The designer has to define the layout of the
modules.
• Encapsulated sensors or modules: the sensors are encapsulated in a physical structure
as modules, and they can be connected together forming a composed interface. The designer
can use the system immediately and has the option of combining the position of the modules.
• Fixed structure: the structure of the system cannot be modified. The designer does not deal
with physical structure.
For the functionality axis, we considered features that help to configure (or program, or map) how
the prototype responds to user inputs. We considered a discrete scale varying from:
• Textual programming: the designer has to program the behavior of the system by code.
• Visual programming: the designer has to program the behavior of the system by
manipulating graphical elements, such as boxes, arrows, or diagrams.
• OSC and MIDI mapping: normally related to the feature that allows the device to be
automatically recognized by the computer, with which the designer has to use an external
software to map the input gestural controller to sound generation outputs.
• Configuration and Mapping using GUI: the system provides a graphical user interface, in
which the designer has to configure input parameters to define how the system reacts to
user’s input.
• Configuration using elements of the physical system (e.g. self-contained mapping): the
system can be modified without the use of external software because it already implements
47
mechanisms in its interface that allows such modification. The designer has to use these
physical parameters to define the behavior of the system.
• Fixed interaction with no configuration: the system presents a limited set of interactions,
and no modification is allowed. Although obtaining immediate results with this approach, the
experimentation is restricted by the pre-defined behaviors. The designer does not deal with
defining functionalities.
Figure 4.7 presents the axes we used to analyze the related tools for prototyping and shows the
trade-off line between flexibility and time-effort to achieve results.
Figure 4.7: Two-dimensional analysis with Functions and Physical Structure axes. In the middle,
the trade-off line between flexibility, and time and effort
Figure 4.8 shows the projects positioned in the axes according to their features. From the broad
spectrum of projects, we considered three major groups. The first comprises tools that require a
certain amount of time and effort to be programmed or to obtain a physical interface. Although
those tools provide wide flexibility, they do not provide rapid results that potentially boost the
cycles of idea exploration. On the other hand, the second group presents a good potential of
delivering fast results, but it is not flexible enough to allow a wide experimentation. Besides, this
low levels of flexibility may limit different contexts of use and possible intentions of the user.
48
Figure 4.8: Analysis of tools which have the potential to be used in physical, functional prototyping
of DMIs. Axis X: Function. Axis Y: Physical Structure
The trade-off spot (highlighted in green in Figure 4.9) may provide a good balance between
flexibility and immediate usability. Besides, the presence of a support opens up possibilities of
gestural explorations. In sum, these systems present a support that holds the modules, can be
configured by means of a GUI or can be used directly as a MIDI or OSC device.
49
Figure 4.9: The green area denotes the trade-off group of tools.
4.2.2. The Trade-off Area
Here, we present a more detailed description of the projects in the trade-off area.
Pin&Play&Perform is a physical interface composed of a set of sensors that can be attached,
removed and reattached, to a surface board (VILLAR; LINDSAY; GELLERSEN, 2005). Each
sensor has a pin that penetrates a multilayer surface, which is responsible for providing energy
and establishing communication with each module. The user can freely arrange the input devices
on the surface. The on-the-fly reconfigurability is similar to our approach. The elements can be
easily placed or removed. The position and the distance between the elements are not restricted
as long as they are on the surface. The diversity of the controls seems to be limited by buttons,
sliders, rotary potentiometers, and encoders. Finally, the surface appears to become limited after
several uses, because of the perforations. This directly affects this device as a tool for rapid and
iterative prototyping, because after a few cycles of experimentation, the surface might have to be
replaced.
Joué is a MIDI controller that consists of a wooden board with a pressure sensor and eight silicon
layers or modules with different forms. The modules can be placed on the board, and the pressure
of user’s fingers and hands is transmitted to the sensor. User’s gestures are constrained by the
shapes present in the layers, which are inspired by drum pads, piano keys, guitar frets. The
modules are held in place by magnets present on its bottom.
50
Mine is a modular MIDI controller based on a support with grid-based slots, on which unit control
modules can be attached. Until this thesis was written, there are six modules: a pad, buttons, a
potentiometer, a rotary encoder, a slider, and a blank module to fill the gaps. The connection
appears to be rigid since it is demonstrated that for removing the module, the user has to use a
special tweezer.
Modulares Interface comprises a frame made of aluminum and plastic, that can be placed on the
top of an iPad screen, and three kinds of aluminum modules: button, slider, and knob. On the
bottom of each module, there is conductive foam, which allows the transmission of the electrical
discharge of the hand to the iPad surface. In sum, the project is an attempt to create a modular,
physical layer to the iPad.
Pin&Play&Perform and Modulares Interface are academic projects that did not become available
publicly. The other two projects are commercial and available for purchase. Mine is strongly
related to the universe of conventional DJ controllers with the possible interactions limited by the
tabletop position and the traditional button-slide-knob paradigm. Although Joué presents a novel
approach that broadens the explorative horizons of DJ controllers, it is based on a pressure
sensor that still presents interactive and diversity limitations.
The trade-off area of the analysis of potential prototyping tools for DMIs presents opportunities
that will be explored further in this work.
4.3. Final Considerations In this chapter, we analyzed DMI literature in search of initiatives that could potentially be used in
the idea exploration phase. We conclude that there are few structured methods and processes
which seem not to focus on guiding the designer in the idea exploration phase.
Also, for prototyping phase, the tools are little integrated, demanding multiple expertise for the
designer to achieve functional prototypes. Besides, considering the specific tools for physical
prototyping, only a few are balanced in the trade-off flexibility and time-effort. The four projects in
the trade-off area present limited diversity of outcomes when we consider structural combinations.
Finally, the relationship between the conceptual frameworks, methods, and processes do not
seem to be integrated with the prototyping tools. This fact produces a gap between idea and
prototype, which can be a hindrance to the cyclic design process.
51
5. EARLY EXPLORATION In this chapter, we present our methodological approach, early exploration actions and the
resulting insights from the overall experience. We attempt to present the genesis of the ideas that
base our proposition.
5.1. Methodological Approach For this work, we followed the design thinking process, which comprises cycles of inspiration,
ideation and implementation (BROWN, 2008), and human-centered design, “that puts human
needs, capabilities, and behavior first, then designs to accommodate those needs, capabilities,
and ways of behaving” (NORMAN, 2013).
We are inspired by agile methodologies which have been widely used in software development
(MARTIN, 2003), with initiatives such as Scrum and eXtreme Programming being well-
established. In other areas, agile principles are also present such as Lean UX, Lean Innovation,
or Lean Manufacturing.
In agile approaches, providing detailed design information does not hold (ADIKARI; MCDONALD;
CAMPBELL, 2009) and the requirements are defined during the course of the project. The
concept of Little Design Up-Front emerges as a pattern that combines user-centered design and
agile development, and the focus is on doing the minimum necessary to bring value to the user
(BERTHOLDO et al., 2014), where the encouragement is to make mistakes fast, often, but early.
Because of that, we decided to follow a spiral, iterative and incremental approach (Figure 5.1),
searching breadth-first instead of depth-first, i.e. developing in small amounts of the whole idea,
instead of focusing on only one aspect of the idea in detail.
Figure 5.1: Our methodological approach: spiral, iterative, and incremental passing through
inspiration, ideation, and implementation phases
Insight A: The user decides what is better for her contexts and intentions Insight B: Focus on quick, iterative and evolutionary process
52
5.2. Project Batebit In the context of developing digital musical instruments for popular music, we took part in a one-
year project in which we conducted explorative interviews with local musicians, developed
preliminary prototypes, evaluated them through interviews, rehearsals and jam sessions.
The project "Batebit: Diálogos entre a Lutheria Digital e a Música Popular Pernambucana"
("Batebit: Dialogues between Digital Lutherie and the Popular Music from Pernambuco") was
funded by the Government of Pernambuco, Brazil, through its cultural fund, FUNCULTURA, and
was conducted by the author of this work, Filipe Calegario, and two other researchers: Jerônimo
Barbosa e João Tragtenberg (BARBOSA et al., 2015a). The project focused on understanding
how a community of popular musicians could adopt new digital musical instruments. Each
researcher was responsible for the exploration of one concept of musical instrument, which was
developed during the research project.
5.2.1. Interviews
Initially, six interviews (Figure 5.2) were conducted with: a Frevo (traditional genre of
Pernambuco) conductor; a DJ with no experience in playing traditional instruments; two musicians
with more than five years of practice; and two musician-luthiers who build their own instruments
also with more than five years of practice. The interviews focused on understanding the
instruments and tools the musicians used in their musical process. Also, trying to understand the
absorption of new technology in their practice. All the interviews were registered on the project's
website (http://batebit.cc).
Figure 5.2: Interviews with Brazilian Northeast popular musicians
A recurrent theme in the interviews was the intimate relationship between musician and
instrument. For instance, one musician mentioned: "I do not think about the gesture I am doing.
It is like a second voice".
The maestro highlighted the possibility of using generic objects, such as rocks and wood sticks,
as musical instruments and reflected how different kinds of musicians could possibly use these
objects in distinctive ways based on previous experience with their own instruments. Besides, the
53
search for alternative ways of expanding techniques with the instrument and explorative approach
towards new uses and new sounds were mentioned as well.
5.2.2. Workshops
The three researchers conducted two workshops of DMI creation with the attendance of twelve
participants in the first one and seven participants in the second one. The participants' profiles
were musicians with interest in digital technology and little or no previous knowledge of the
prototyping tools.
The first one (Figure 5.3) covered three tools for prototyping instruments: Arduino, an electronic
prototype platform; Pure Data, a visual programming language for sound manipulation; and
Ableton Live, a digital audio workstation with real-time control of processes, mainly used by DJs.
Figure 5.3: First workshop of DMI creation
Due to the complexity of covering a broad spectrum of topics, the researchers decided that the
second one (Figure 5.4) would cover only the Arduino platform and its capabilities of producing
sound without other tools.
Insight C: Leverage the existing intimate relationship musician-instrument to conceive new instruments.
54
Figure 5.4: Example of one of the instruments created with Arduino during the second workshop
In both workshops, the researchers showed the content in a hands-on approach, presenting
examples and encouraging the participants to experiment the use the tools for musical
expression. Despite this didactical effort and the availability of the tools as hardware platforms
and open-source software, the workshops showed that technical details were perceivably an
initial barrier for most people, who took a long time to achieve a musical result (in some cases
not even reaching it).
5.2.3. Pandivá
Following these insights, we developed an exploratory functional prototype which merged the
guitar-inspired posture, the way of triggering sounds by tapping a tambourine skin and the way of
altering the pitch using a trombone slide. The instrument was called Pandivá (reduction of
Portuguese words "pandeiro de vara", in English: slide tambourine) (Figure 5.5).
Figure 5.5: Pandivá #0, Pandivá #1, Pandivá #2.
For the first functional prototype (Pandivá #0, Figure 5.5), we used piezoelectric sensors inside
rubber layers as pads for the player to strike and a slide made of concentric pipes of PVC with
different diameters. In one tip, we placed a small LED and the other, a light-dependent resistor.
We connected the sensors to an Arduino Uno that sent MIDI messages to a computer, which was
Insight D: Encapsulate technical details to allow the users to reach a musical experimentation faster and with less effort.
55
responsible for synthesizing the sound. With this version, we could test sensor combinations and
possible mappings.
The second functional prototype (Pandivá #1 in Figure 5.5) incorporated a structure made of
laser-cut MDF. The instrument could be used on the lap or with a shoulder strap. This version
comprises fifteen contact buttons placed in three groups, used to detect a strike on its surface
and trigger notes. The groups of buttons attempted to represent the playable areas in the pandeiro
skin surface. Four piezoelectric sensors measure the intensity of the attack, and a slide
potentiometer coupled to the sliding PVC pipes form a piston that controls pitch.
The player triggers notes by striking on one of the three sets of buttons located on the instrument
body and alters their pitch by moving the slide. Using a mode button, the player could switch
between melodic mode, which triggers notes with pitch based on slide position; and harmonic
mode, in which the slide changes the chord that is being played and the three pads correspond
to the three notes of the chord.
The sound of Pandivá #1 was synthesized on the computer by GarageBand via a MIDI
connection. We chose the steel string guitar as the primary instrument for demonstration, due to
its evident attack that exemplifies the rhythmic characteristics of the device.
After testing this version of the Pandivá prototype with four users (Figure 5.6), three of them
mentioned that the way of altering the pitch could be improved to allow a discrete control because
managing to reach a specific note was difficult with the slide.
Figure 5.6: Testing Pandivá with three percussionists and a guitar player
To provide this functionality, we developed a new version of Pandivá (Pandivá #2) that had a set
of four buttons instead of a slide. Combining the buttons, the player could trigger up to sixteen
different notes. Both versions of Pandivá can be seen in Figure 9.
56
The Pandivá #2 was developed and tested with two performers. As soon as a left-handed
musician used it, he felt uncomfortable with the buttons positions. To solve this issue, we made
additional holes on the opposite side and moved the buttons to the left-handed position. The
adapted version can be seen in Figure 5.7.
Figure 5.7: Pandivá version 2 modified for left-handed users.
An overall reflection is that, for all the modifications, we had to switch our development mindset
several times from structure, electronics, programming, and sound design. Time and effort we
have expended while changing modes of operation have been a hindrance in our iterative design
process.
5.2.4. Sandbox Wow
Sandbox Wow was a prototype developed during Batebit project to experiment possible musical
interactions with: (1) eight surfaces made of a homemade conductive ink with graphite powder
and white glue; (2) eight pieces of a capacitive sensor with Arduino digital ports; and (3) eight DIY
sensitive pad made with a sandwich of two sides of EVA rubber and a piezo as filling. It could
function as a sequencer or a MIDI controller.
In the project, as Figure 5.8 shows, we had to deal with different development contexts such as:
structure (mechanical support, mechanisms, material choice), electronics (electronic
components, sensors, actuators), programming (communication protocol, coding), mapping
(operations, scaling, adaptation, connections), sound (synthesis, parameters choice, content
Insight E: Allow the user to dynamically modify the functional prototype to adapt it for her contexts of use and intentions.
57
choice). For each context, we used specific sets of tools, which required a spectrum of different
expertise. Besides, the often shifts in mindsets may have interfered in the cognitive load, since
these shifts spread the attention to aspects of different natures of the object or its behavior.
Figure 5.8: Example of different contexts (mechanisms, electronics, programming, mapping,
sound) to build a functional prototype of a DMI (Video can be seen: http://youtu.be/l2HnE3txKdc)
Insight F: There are multiple contexts during the development of a DMI functional prototype, which demand different kinds of expertise.
58
5.3. Summary The early exploration allowed us to collect relevant insights for the development of this project.
Here is the summary of the insights:
• Insight A: The user decides what is better for her contexts and intentions
• Insight B: Focus on quick, iterative and evolutionary process
• Insight C: Leverage the existing intimate relationship musician-instrument to conceive new
instruments.
• Insight D: Encapsulate technical details to allow the users to reach a musical experimentation
faster and with less effort.
• Insight E: Allow the user to dynamically modify the functional prototype to adapt it for her
contexts of use and intentions.
• Insight F: There are multiple contexts during the development of a DMI functional prototype,
which demand different kinds of expertise.
59
6. PROPOSITION In this chapter, we explain the decisions we made in terms of scope and present our proposition
that consists of a concept, a method, and a toolkit.
6.1. Scope and Basis Based on literature, in Figure 6.1, we present the scope that we decided for this work. Firstly, we
choose to explore the implementation of gestural controllers and to prepare them for mapping.
We aim to provide tools which aid the design of artifacts that would lay in the border between
instrument-inspired and alternate gestural controllers.
Furthermore, due to previous experience in the area, we will focus our efforts on manipulative
gestures, i.e. physical objects, and we will not deal with empty-handed or free gestures. We
expect that our approach will aid designers and performers to obtain exploratory prototypes,
focusing on idea exploration and prototyping phase of the DMI design process. Besides, we aim
to provide ways of implementing functional prototypes, which means that we are not concerned
to assess either appearance or the final product.
Figure 6.1: Scope of this work
Our proposition comprises a concept, a method, and a toolkit. In this chapter, we present each
point describing the cumulative relationship they have, i.e. the toolkit contains the method that
contains the concept (Figure 6.2).
60
Figure 6.2: Three elements of our proposition
We propose to explore the concept of instrumental inheritance, to adapt a design method for idea
generation, called morphological analysis, to be used in the DMI context, and a physical
prototyping toolkit based on the morphological chart.
6.2. Instrumental Inheritance Inspired by the Insight C (“Leverage the existing intimate relationship musician-instrument to
conceive new instruments”) and by the discussions in our research group on Music, Technology,
Interactivity, and Creativity (MusTIC) at Centre for Informatics - UFPE, Brazil, we started to
develop the concept of instrumental inheritance, which is the similarity in shape or playing
techniques a certain instrument transfers from other existing instruments. MusTIC’s discussion
covered a hypothesis that instruments with higher levels of inheritance have more chances to be
adopted due to the existing body of knowledge and gestures repertoire. Our understanding is that
instrumental inheritance is composed of two other inheritances: structural, physical elements of
the instruments, and gestural, ways of manipulating the instrument.
We propose the use of instrumental inheritance as an initial constraint to ignite the creative
process for new instruments ideas. Though the new resulting instruments will not be structurally
or gesturally restricted by acoustic laws, this approach will serve as a kick-start method to explore
and generate ideas based on common knowledge and existing cultural hooks, serving as an
initial, structured and exploratory path for idea generation. This decision is supported by idea
generation literature which defines the constraints and the use of analogies and metaphors as
idea generation promoters (SHAH; KULKARNI; VARGAS-HERNANDEZ, 2000).
6.2.1. Related Concepts
In literature, there are some related concepts that do not exactly describe the specificities we
attempt to communicate with “instrument inheritance”.
Related to the discussion, skeuomorphism is a concept that designates the incorporation of
elements of existing artifacts in new artifacts, even not presenting a functional importance for the
new one (NORMAN, 2013). It is originated in the field of archeology, where researchers call a
61
skeuomorph “an element of design or structure that serves little or no purpose in the artifact
fashioned from the new material but was essential to the object made from the original material”
(BASALLA, 1989). O’Hara (2012) mentions that the original concept of skeuomorphism can be
observed but not intentionally induced (O’HARA, 2012). That is why the author strongly disagrees
with the use of this concept in the context of user interface design, which is commonly adopted
by Apple to describe elements of interface based on the aesthetics of physical objects (such as
the bookshelf in iBooks) (CURTIS, 2015). According to O’Hara (2012), Apple is using visual
metaphors and not skeuomorphism.
In its turn, a metaphor is a conceptual tool used to interpret aspects of an idea or artifact based
on a comparison of the characteristics of another idea or artifact (HEY et al., 2008). It is a concept
that appears related to analogy, which can be understood as the transfer of information from
familiar, existing domain in order to explain or define elements of a different domain (DAHL;
MOREAU, 2002). In design, metaphors and analogies are commonly used to generate new ideas
based on existing ones (HEY et al., 2008). In DMI design, the use of metaphors is related to more
transparent communication between the performer and the audience yielding a more expressive
instrument (FELS; GADD; MULDER, 2002).
The three concepts can be summarized as a transfer between characteristics of an existing
artifact to another (in the case of skeuomorphism, observable but not intentional). In our case,
instrument inheritance defines elements that are passed or transferred from a predecessor to a
successor (MERRIAM-WEBSTER, 2004).
Other related concepts that we will use to refine ours is affordance and signifier. According to
Tanaka et al. (2012b), “affordances are a configuration of properties that provide a direct link
between perception and action.” (TANAKA; ALTAVILLA; SPOWAGE, 2012). The term was
defined in the context of psychology by James Gibson, who tried to present an alternative
ecological discussion about visual perception (KAPTELININ, 2014). For Gibson, affordances are
“action possibilities offered by the environment to the animal” (GIBSON, 1979), or as interpreted
by Norman (2013): “the physical objects conveyed important information about how people could
interact with them, a property […] named “affordance”’ (NORMAN, 2013). According to Norman
(2013), even if it is not visible, the affordance is present between the environment and the user.
Thus, it is important for the designer to signal how the components of an artifact should be used
or how they work to transform the invisible affordance into perceived affordance for the users.
Norman (2013) highlights that perceived affordances aid people to realize possible actions
without recurring to labels, marks or instructions (NORMAN, 2013). The author presents the
concept of signifier as “any perceivable indicator that communicates appropriate behavior to a
person. Affordances determine what actions are possible. Signifiers communicate where the
action should take place.” (NORMAN, 2013).
With our concept of instrumental inheritance, we aim to provide new instruments with signifiers
inspired by existing instruments. In this way, it seems to be possible to leverage a current body
of knowledge, playing techniques, and familiarity. For the audience, inheritance can provide
62
familiar elements which can possibly avoid a strange reaction and disconnection, causing
engagement, and perhaps more expressiveness.
6.2.2. Possible Evidences
In an attempt to illustrate the instrumental inheritance, we discuss possible cases where the
concept can be applied.
We can speculate that the success and popularization of Moog Synthesizers were related to the
implementation of an existing piano standard keyboard. On the other hand, Buchla Synthesizers,
which implement an alternative controller, are known for being used by a niche of experimental
musicians (FANTINATTO, 2014) (TEBOUL, 2017). However, it is important to highlight that there
are different factors and variables involved in instrument adoption, which we consider a complex
issue, and this speculation is one possible view in an attempt to understand the context.
In the same direction, we can conjecture that electric guitar players took advantage of the existing
familiarity, body of knowledge and playing techniques of acoustic guitars because of the inherited
instrument shape and ways of controlling (six strings, selecting and plucking the string with a
hand, and selecting the string and the fret with the other hand). It does not mean that the electric
guitarists remained imprisoned by playing techniques of acoustic guitar, but they have possibly
started from a common ground and developed other paths. It is different from a completely new
instrument which does not convey familiar information on how it works or how it can be used.
Following a similar trend, another possible evidence is related to the keyboard layout that was
transferred from instruments such as clavichord and harpsichord to the piano (SACHS, 1940).
6.2.2.1. Hybrid Instruments
We consider that hybrid instruments inherit from more than one musical instrument, combining
ways of holding, ways of playing, or shape. Looking through the lenses of instrumental
inheritance, instruments like the keytar (Figure 6.3) seems to benefit from the body of knowledge,
and the gestures repertoire from both keyboard and guitar.
63
Figure 6.3: James Brown playing a Moog Liberation keytar. Source: synthtopia.com
As Figure 6.4 shows, other examples are the Scratchocaster, developed by Viny’ Lourd Son,
merges shape of a guitar with a mixer and a turntable; the DRUMITAR, developed by Futureman,
member of the band Béla Fleck The Flecktones, guitar-inspired body for a drum set;
KIMOPHONE, developed by Kimo Lobo, which is a series of instruments that hybridize a sax-like
mouthpiece with a keytar; and Arduino Ribbon Synth, a DIY project by Dean Miller, that
implemented a drum pad and a fretless-inspired ribbon sensor that is hold like a guitar.
Figure 6.4: (a) Scratchocaster, (b) DRUMITAR, (c) KIMOPHONE, (d) Arduino Ribbon Synth
Furthermore, an example of a commercial project that incorporates the concept of instrument
inheritance is the Artiphon (Figure 6.5), which allows the user to hold the instrument in positions
64
inspired by cello, violin, keyboard, and guitar. The controllers are inspired by piano’s keys, guitar’s
strings, and frets.
Figure 6.5: Artiphon
The modulin (shortening for modular synth + violin) is an instrument developed by Martin Molin
(member of the band Wintergatan) that consists of a ribbon sensor connected to a modular
synthesizer. It is held and played like a violin (Figure 6.6).
Figure 6.6: Modulin, instrument developed by Martin Molin integrant of the Swedish band
Wintergatan
6.2.3. Discussion
One possible criticism about basing the development of new instruments on existing acoustic
counterparts is the difficulty of avoiding clichés, as the majority of playing techniques and search
for new sounds have already been explored (MAGNUSSON; MENDIETA, 2007b). In a survey
with musicians, Magnusson and Mendieta (2007) discussed that some participants negatively
mentioned the DMI that is “slave of the historical”, which can diminish its potential to be an original
65
and creative way to expand musical expression. For that, we argue that our intention of using
instrumental inheritance go in the direction of using the familiarity as inspiration, and leverage
existing cultural hooks. Considering the DMI classification, Miranda and Wanderley (2006) made
an important distinction between instrument-like and instrument-inspired gestural controllers
(MIRANDA; WANDERLEY, 2006). As mentioned before, we focus our work on inspiration. We
speculate that there is a creative potential on exploring combinations of instruments parts, and
we consider to use the inheritance to ignite initial ideas to be explored depending on the context.
6.3. Morphological Chart for DMI Idea Generation Gathering the insights from previous sections, here, we explore the potentials of combining
elements of existing instruments to generate ideas of new instruments. For that, we followed a
well-established design method called morphological analysis. Shah et al. (2000) classify
morphological analysis as a germinal method, i.e. methods used in the initial step of conception
when the designer has yet no previous ideas. This characteristic is strongly related to our first
question, making the morphological analysis suitable for our use. Besides, this method provides
a quick way of visualizing and combining existing parts.
6.3.1. What is Morphological Analysis?
The morphological analysis is an idea generation method in which existing artifacts are split into
their fundamental parts and then recombined to generate new ideas (RITCHEY, 1998). The
method was firstly proposed by Fritz Zwicky, a Swiss astrophysicist, and aerospace scientist, in
the context of generating alternatives for jet propulsion (ZWICKY, 1967). He analyzed and split
the propulsion system into six functions, and through combination, he demonstrated he could
achieve more than five hundred possible alternatives for jet propulsion design (VASCONCELOS
et al., 2016).
According to Smith (1998), the morphological analysis can be described as an analytical strategy
based on decomposition, in which wholes are divided into parts or attributes, and ends into means
(SMITH, 1998). The author states that this approach was the most often used in his analysis.
Cross (2000) describes the procedure to formulate a morphological chart (or matrix, table, box)
(CROSS, 2000) as:
1) “List the features or functions that are essential to the product
2) For each feature or function list the means by which it might be achieved
3) Draw up a chart containing all the possible sub-solutions
4) Identify feasible combinations of sub-solutions”
The advantage of using a morphological analysis approach is to have an overall picture of the
possible solution space that can be explored in a structural and systematic way. According to
Vasconcelos (2016a), this systematic approach consequently “forces designers to consider many
potential solutions that would otherwise be overlooked” (VASCONCELOS et al., 2016).
66
As said by Pahl et al. (2007), the problem of this method is to select which combinations are
compatible (PAHL et al., 2007). Besides, Cross (2000) mentions that depending on the number
of combinations; the search can become impossible (or really tedious) (CROSS, 2000). For that,
the author suggests discarding some incompatible combinations up-front, or taking a random,
intuitive approach to choosing the possible paths.
To illustrate how the morphological chart is implemented, Figure 6.7 shows a generic
representation of a morphological chart and Figure 6.8 presents a morphological chart for forklift
trucks.
Figure 6.7: Representation of Morphological Chart (extracted from Pahl et al. (2007))
67
Figure 6.8: Example of a morphological chart for forklift trucks (extracted from Cross (2000))
6.3.2. Morphological Chart Based on Instrumental
Inheritance
Here, we follow the morphological analysis steps described by Cross (2000) (CROSS, 2000) to
produce a morphological chart based on instrumental inheritance.
1) Features or functions that are essential to the product
For an acoustic instrument, an essential feature is the physical structure, not only for producing
the sound (as we are not interested here) but also for providing a support that can be held, on
which the hands can perform the three functions of the gestural channel: ergotic, semiotic, and
epistemic, and elements of manipulative gestures.
Considering instrument classification as an inspiration source to extract instrument’s features,
Hood (1982) criticizes the limited focus of the organology (or the science of musical instruments)
being practiced at the time on the description of physical features and acoustic properties of the
instrument, disregarding points such as techniques of performance, and musical functions.
Because of that, his approach to classification differs from established ones, such as (KNIGHT,
2015) and (MIMO CONSORTIUM, 2011), whose primary focus was on how the material
classification of the instruments (e.g. how the instrument is made or how it acoustically generates
sound), mainly used to organize the plethora of historical instruments in museums. The author
proposes elements that focus on the relation between the performer and the instrument, such as
the instrument support, which is the “manner in which the instrument is supported” (HOOD, 1982).
68
In the context of analyzing the appropriation and the emergence of stylistic variation between
different performers, (ZAPPI; MCPHERSON, 2014) and (GUREVICH; MARQUEZ-BORBON;
STAPLETON, 2012) present features that were the analysis results of the use of an overly
constrained musical interface. The features related to gestures are affordances, interaction
techniques, ways of holding, and ways of playing.
Inspired by these initiatives, we decided to focus on features that are related to instruments:
• Physical structure: focusing on the way the object induced player's postures, the way
the player holds or the object is supported.
• Gestural control: we are interested in the player's instrumental gestures to control
sound.
2) Means by which each feature or function might be achieved
Considering the physical structure, Figure 6.9 shows artifacts that are intentionally represented
as generic blank objects to highlight the ways of holding and the posture they induce when used.
Figure 6.9: Postures inspired by existing instruments. Drawings by Giordano Cabral
For our initial set, we explore supports with signifiers based on popular instruments. The postures
were inspired by guitar, tambourine, accordion, drum pedals, flute/saxophone/clarinet, cájon, and
piano (Figure 6.10).
69
Figure 6.10: Simple analysis of existing musical instruments associated with an abstract symbol of
control
Regarding ways of control, Figure 13 show elements inspired by the hurdy-gurdy crank, the DJ
turntable, the kalimba tines, the percussion instruments skin, the violin bow, the trombone
mouthpiece, the guitar strings, the trombone slide, and the piano keys. Although the inspiration
comes from existing instruments, we depicted the controls as simple generic drawings aiming to
70
present them in a more abstract way and broaden their uses beyond the existing instrument. We
focus on controls of instruments we considered that have potential to expand combinations.
3) Chart containing possible sub-solutions
4) Feasible combinations of sub-solutions
The current version of the chart comprises seven postures and nine controls organized in three
groups of control (excitation, modification, and selection) according to the instrumental gestures
classification by (CADOZ; WANDERLEY, 2000) (Figure 6.11).
Figure 6.11: Chosen path on the morphological chart: a mix of guitar-based posture, a slider, and
pads. Drawings by Giordano Cabral
6.4. Development of the Functional Prototyping Toolkit for DMI
In this section, we will present the development cycle of the proposed prototyping toolkit for DMIs,
the Probatio (the Latin word for "test, experiment, trial").
The morphological chart is a useful tool to visualize and navigate in the design space of musical
instruments, but as mentioned before, prototypes are a crucial part of the design process and, for
a better understanding of a DMI, functional prototypes are more suitable. As we aim to boost the
design cycle of DMIs, it is important to have the functional prototype, so the user, or designer,
can easily modify and evaluate the generated idea.
Concerning the process of implementing functional DMI prototypes, we decided to build a
prototyping toolkit that will work as a physical morphological chart based on instrumental
inheritance.
The toolkit approach seems promising as presented by (SADLER et al., 2016a) and (HELMINEN;
AINOA; MÄKINEN, 2015), which respectively discuss the benefits of encapsulating technical
71
details to achieve more creative results, and the importance of toolkits in transferring knowledge
from the user to the developer to achieve adequate alternatives during the design process.
With our prototyping toolkit, the user can change the controls and postures by using and
combining physical modules that are already responsive. Inspired by the related projects, the
basic idea is to build a system that has (1) blocks, the units of control that enable the gestures of
the player to produce sound, and (2) supports, in which blocks can be placed.
The objective of the prototyping toolkit is to provide an initial, structured and exploratory path for
the designer, or performer-designer, to generate instrument ideas and to reduce time and effort
to build functional prototypes.
6.4.1. Guidelines
Based on the insights that were collected from the literature review and the early exploration, we
formulate our design guidelines which help our decision making during the course of the
development.
• Tangibility: the toolkit should present tangible modules to relate to the “strong need for a
physical connection with their instrument.” (PAINE, 2013).
• Modularity: the toolkit should allow the user to explore combinations of a diverse set of
modules (MAESTRACCI; FRECHIN; PETREVSKI, 2011) and to easily perceive causal
properties of connections (GELINECK; SERAFIN, 2010a). This guideline is inspired by
Insight A (“the user decides what is better for her contexts and intentions”) and Insight E
(“allow the user to dynamically modify the functional prototype to adapt it for her contexts of
use and intentions”).
• Technical Encapsulation: the toolkit should encapsulate technical details to reduce time
and effort to build prototypes, allowing the user to focus on creative thinking (SADLER et al.,
2016b). Directly, the toolkit should “expose the functionality and abstract the underlying
technological complexity” (KNORIG, 2008). This guideline is inspired by Insight B (“focus on
quick, iterative and evolutionary process”), Insight F (“there are multiple contexts during the
development process of a DMI functional prototype, which demand different kinds of
expertise”), and Insight D (“encapsulate technical details to allow the users to reach a musical
experimentation faster and with less effort”).
Besides, we defined additional requirements that we believe will enhance the usability and the
users’ experience with the environment:
• Feedback: the environment should provide a clear, perceptible and real-time response to
actions and modifications performed by the users (JORDÀ, 2003) (O’MODHRAIN; CHAFE,
2000).
• Integration: the environment should be easily integrated with previous or legacy systems,
benefiting from their functionalities (SCHMEDER; FREED, 2008).
72
6.4.2. Implementation Decisions
To help to fulfill the requirements, we took the following implementation decisions.
• Build the environment using techniques of digital fabrication: use techniques of digital
fabrication to obtain tangible artifacts with a better accuracy and quality with less time and
effort (WALTER-HERRMANN; BÜCHING, 2014) as experienced during the exploratory cycle.
• Use of open-source electronic prototyping platforms: use established open-source
platforms, such as Arduino (BANZI, 2009), and sensors that are available and easy to
integrate due to increasing trend of DIY and Maker Movement (DOUGHERTY, 2012).
• Implement standard protocols: use standard protocols such as MIDI and OSC (WRIGHT;
FREED; MOMENI, 2003) and leverage existing systems for mapping and development
ecosystems such as libmapper (MALLOCH; SINCLAIR; WANDERLEY, 2014). With that, we
hope to reduce the time and effort for the sound production.
6.4.3. Physical Structure
As a preliminary validation, we prototyped a nonfunctional version of the system with MDF pieces,
and hooks and loops fastener (Velcro). The supports (pieces (a), (b), (c), and (i) in Figure 6.12)
had the hooks, and the control parts ((d), (e), (f), (g), and (h)) had the loops. With this early proof-
of-principle, one could already understand the potential of trying different combinations and
understand how the system would work. This prototype was important to communicate the idea
to potential users and to have a better grasp about the desired dimensions of the final toolkit.
Figure 6.12: Initial set of nonfunctional modules made of MDF and Velcro: (a) guitar-inspired body, (b) clarinet-inspired body that can also be a guitar-inspired neck, (c) tambourine-inspired body, (d)
73
guitar fretless neck inspired input, (e) tambourine-inspired three pads input, (f) guitar-inspired strings input, (g) wind instrument-inspired breath input, (h) piano keys-inspired keyboard, (i)
combination of part (a) and part (b) resulting in a guitar-inspired support, and (j) combination of inputs and guitar-inspired support.
In order to increase the granularity choice of possible positions to place the control blocks, we
decided to build slots in the supports that are presented in Figure 6.13.
Figure 6.13: Sketches of Probatio's supports
To visualize how the necks and bodies would fit together, we rendered 3D models as shown in
Figure 6.14. This step was important to realize how the neck would be connected to the body,
and how many slots would be the minimum necessary to implement functions following the
modularity guideline.
Figure 6.14: 3D renderings of Probatio’s supports and block
Aiming to obtain quicker results, we decided to build the structure with laser cut MDF (medium-
density fiberboard) 3mm-thick. MDF is an engineered wood made of residuals wood fibers, mixed
with wax and resin, and formed with high temperature and pressure. It is a plain and rigid material
with an inexpensive cost (MALONEY, 1996).
The rapid prototyping technique based on laser cutter favored our development process because
it is simple to model since it is basically based on 2D drawings, provide precise cuts, and a short
74
time to achieve the product. With the proliferation of the Maker culture, the cost of laser cutting is
decreasing, and the availability of machines is growing.
To build our tridimensional structures, we used a technique based on joining two pieces of wood
by cutting a set of matching rectangular cuts, called finger joints. This kind of joint makes the
connection between two boards stronger and easier to build since it helps to keep the two boards
in place and then glued together.
We used a commonly used web tool in the Maker community to generate the finger joints, the
MakerCase website (Figure 6.15). The website presents a simple interface where the user
provides the dimensions of the block, and it generates the basic 2D cutting plans.
Figure 6.15: MakerCase.com was used to generate the finger joints of the blocks
After processing the drawings in Adobe Illustrator to add the desired details, we produced, for
instance, the cutting plans of a four-slots support for Probatio (Figure 6.16).
75
Figure 6.16: Example of drawings used for laser cutting
To check if the plans were correctly generated, we imported the 2D drawings and modeled in 3D
using Dassault SolidWorks (Figure 6.17). This step was important to avoid rework because of
errors, besides reducing the wasted material and time.
Figure 6.17: 3D rendering for checking if all the components fit together before sending to laser
cutting
6.4.4. Connection Slots
Following our modularity and technical encapsulation guideline, we decided to expose the
functionality of the controllers but hide the technical details of blocks.
The blocks can be positioned on the supports by inserting them into the slots. The connection is
made through spring-loaded pins in contact with a metal surface (Figure 6.18).
76
Figure 6.18: The spring-loaded connector
To ensure that the block is held in place and also to increase the adhesive force, we used four
neodymium magnets placed in each slot, and the same amount placed in each block. These
magnets are stronger than the ferrite counterparts, but also more expensive. In our case, as the
blocks have restricted room within, we decided to reduce the needed volume for the components
by choosing neodymium magnets.
Our first attempt to build the metallic contact surface for the slots was based on etching copper
plates, a common technique for building a circuit board. The process involves drawing on the
copper board with a permanent marker and submerge the plate in a solution of ferric chloride
(Figure 6.19). The surface without the marks dissolves, keeping the metallic paths intact that will
serve as an electronic contact. The procedure was laborious and with a high risk of affecting the
needed accuracy for placing the blocks. Besides, this way of building was not modular,
constraining the possibilities of building supports with different shapes.
Figure 6.19: Building the contact slots by etching copper plates
77
As a better and modular alternative, we developed the slot module using a CAD for printed circuit
board (PCB) and order a few boards in a Chinese PCB manufacturer. The main disadvantage is
the time it takes for the package to arrive, but the gains are in quality and accuracy. With the PCB
unit-slot, it was only a matter of soldering and arranging the boards in the support, as can be seen
in Figure 6.20.
Figure 6.20: Connector based on printed circuit designed in EAGLE and manufactured in China
6.4.5. Blocks
Each block has its microcontroller (the slaves) which communicates with the central hub (the
master) through a wired connection and uses the I2C (Inter-Integrated Circuit) serial
communication bus, a master-slave-based protocol whose connection consists of four wires:
VCC, GND, SCA, SDL. Amongst other buses, such as SPI, CAN, UART, the benefit of using I2C
is the multi-slave with only two-wire per slave, i.e. there is no need of a dedicated wire connection
for each slave or additional circuitry. This reduces the number of wires to connect and becomes
simpler and cheaper to implement this prototype. Figure 6.21 illustrates the connection between
the block and the slot. The GND and VCC bus provides the electricity for the block to work, SCL
is the clock line, and SDA is the transmitted data.
An alternative option for communication and power supply was to transform each block in an
autonomous, battery-powered module that would wirelessly connect directly to the computer and
would eliminate the hub from the architecture. We considered that the management of batteries
charge would bring a layer of complexity that could negatively affect the time of our development.
Simply put, it would take some time to recharge the batteries, and it would take longer to develop.
Therefore, we chose the wired-block approach.
78
Figure 6.21: Block and slot’s I2C connections
The main loop of the central hub is illustrated in Figure 6.22. Firstly, the hub checks all the
registered slave address. If the slave replies, it means the block is connected to a slot. If the slave
does not reply, it means the block is not present in the support.
The hub gathers all the sensor values of each block and format all the data in one message.
Then, it sends this message to the computer via the serial port following the protocol which is
shown in Figure 6.23. The position of values in the message are fixed. If a block is not present,
the hub fills the position in the message with 0.
Figure 6.22: Sequence diagram of communication between the hub, the blocks, and the computer.
Between the hub and the blocks are I2C messages.
This approach guarantees that the size of the message remains constant, even with the varying
number of blocks in the support. This avoids irregularities in the time each message takes to
reach the computer, reducing jitter, which is the deviation of a periodic transmission. We
performed an informal test to assess the latency of the system, and the result was approximately
5 milliseconds.
79
Figure 6.23: Communication Protocol used between Probatio's hub and the computer via Serial
Port
Software written in Java receives the serial messages and plot the values of each connected
block for monitoring purposes. Probatio’s overview is illustrated in Figure 6.24.
Figure 6.24: Probatio's communication overview. The supports are highlighted to show they are simply an extension of the physical communication bus of I2C. The logical parts are the blocks
which communicate to the hub.
80
6.5. Final Considerations In this chapter, we presented our threefold proposition that addresses our research questions.
For the structured and exploratory steps for idea generation, we propose the use of a
morphological chart based on the concept of instrumental inheritance. As we aim to boost the
cycles of idea exploration and prototyping, the morphological chart appeared to be distant for the
user to experiment the functional instrument. Therefore, we decided to incorporate the
combinatory nature of the morphological chart into a toolkit for physical, functional DMI
prototyping. In this sense, the toolkit embeds a method, which already embeds a concept.
Following our guidelines, the physical toolkit was implemented to encapsulate technical details in
the format of modules that can be combined in different ways to achieve the intended user’s
result.
81
7. EVALUATION OF PROBATIO 0.1 In this chapter, we present a description of the version 0.1 of Probatio and perform an exploratory
evaluation (CALEGARIO et al., 2017).
7.1. Description The first functional prototype of Probatio comprises five blocks and two bases (for an online video
demonstration: https://youtu.be/Ge_aj5uMgOU). As can be seen in Figure 7.1, the first support is
inspired by the player’s posture when playing the piano, also known as a tabletop posture, and is
laid out in a three by three square grid. The second support is inspired by the posture of the
performer when playing the clarinet or the saxophone and it has four slots in a single row. Each
slot can fit a block either from the top or sides, allowing multiple block orientations (except for the
center slot in the three by three grid). As an example, the 4-slots and 3-by-3 supports can be
combined to create a compound support that resembles a guitar body and neck.
Following the support options mentioned by (HOOD, 1982), the bases and their combinations can
be used on a stand or table, player's lap, between legs, across legs, hold by hands.
Figure 7.1: Probatio's bases
As a feasible alternative that we believe would not compromise the concept of the environment,
we developed the following five blocks (Figure 7.2):
• Fretless: inspired by fretless necks of string instruments. It is made of a resistive touch tape
and a force-sensing resistor and measures the position and force of the finger on the surface.
• Turntable: inspired by DJs' turntables. It is made of a circular piece of MDF attached to a
rotary encoder.
• Bellows: inspired by the bellows of a harmonium. It is made of a moving top connected to
the body of the block by a central pin and a pair of springs. It has a small magnet on the
moving top and a hall effect sensor on the internal wall of the body. As the top is depressed,
the distance between the magnet and sensor changes, which indirectly allows the rotation of
the top to be measured.
• Buttons: inspired by discrete controls in several instruments such as accordions' bass
switches, piano keys or brass instrument valves. It comprises four buttons.
• Crank: inspired by the crank of the hurdy-gurdy. It consists of a crank attached to a rotary
encoder.
82
Figure 7.2: Probatio's blocks
Figure 7.3 presents a possible combination using the five blocks and a compound base.
Figure 7.3: Possible combination of a DMI prototype using Probatio
As can be seen in Figure 7.4, combining the supports in different positions and orientations
provides diverse ways of holding Probatio, inducing various postures.
Figure 7.4: Example of possible postures using Probatio
7.2. Evaluation In this section, we present a preliminary evaluation of the system. We consider this first cycle of
evaluation as the first opportunity to collect initial impressions about Probatio’s shape and
functionalities.
83
7.2.1.1. Objective
We conducted a preliminary study to collect feedback on Probatio and its intrinsically associated
design method. The main purpose is explorative.
7.2.1.2. Participants
We enrolled 10 volunteers (3 women and 7 men), ranging in age between 24 and 50 years old,
with one of them having no formal musical training. All of the participants had experience using
digital technology in music, and nine had already developed at least one DMI and were familiar
with the development of digital technology for music.
7.2.1.3. Design
The sessions (Figure 7.5) were performed with one participant at the time and lasted
approximately one hour each. The participants were first introduced to the prototyping toolkit and
asked to explore it freely for 10 minutes as a practice session. After that, the participants were
instructed to perform a set of five simple tasks: 1) "insert a block in the side slot and play with it",
2) "make the movement of a block interfere with the sound of another block", 3) "use more than
two blocks at once", 4) "make a compound base", and 5) "hold the base in a different way". The
tasks were independent of each other, and they were given following the same order for all the
participants. For each task, we notified the participant as soon as we observed that he or she
achieved it.
We chose these 5 tasks since (a) they are easily observable, (b) they are likely to be performed
consistently by all the participants whatever their skills in the design of DMIs, and (c) they are
representative of the basic assembly tasks that can be performed with the current version of the
toolkit. Moreover, the tasks acted as a stimulus for the participants to understand the features of
the system and then enable a proper discussion about it.
Following this, the participants took part in a semi-structured interview covering their impressions
about the current version of the system and an overall discussion about the prototyping toolkit
concept. The interviews were video recorded, transcribed, and analyzed according to the thematic
analysis method used in (TANAKA et al., 2012).
84
Figure 7.5: Participants experimenting Probatio
7.2.1.4. System Setup
To focus on the structure and use of the physical prototyping toolkit, we decided to exclude the
possibility for the participants to create their own mapping with a computer. Though one-to-one
mappings have been shown to be a limiting factor in DMI design (HUNT; WANDERLEY;
PARADIS, 2003), we consider that they are effective in the context of this experiment. Therefore,
each Probatio block was uniquely mapped to a sound, generated via Musical Instrument Shield
85
for Arduino by Sparkfun. A demo video of this implementation can be seen at
https://youtu.be/Ge_aj5uMgOU.
7.2.1.5. Analysis
In this section, we grouped the participants' responses into four major groups: (1) positive points,
(2) negative points, (3) suggestions, and (4) contexts for use. For each of these groups, we
present the main points raised and quotes from the interviews with the indication of the participant
code in parenthesis.
7.2.1.5.1. Positive Points
The most recurrent positive points during the interviews were: (1) the physical affordances that
invite to use the system (mentioned by 50% of the participants). Illustrating this, one participant
said, "They [the blocks] just suggest: pick me up and do something with me" (P01); (2) the clarity
of the way it works (40% of the participants). For instance, one participant mentioned: "A cool
thing about it is that it is self-explainable" (P02), and two participants highlighted that they felt the
system to be "quite intuitive" (P03, P04); and (3) the convenient and functional connection
between the blocks and the bases (30% of the participants). For example, a participant
highlighted "I think it is brilliant" (P05) and another participant described the connection as "pretty
solid" (P06) since the magnets held the blocks in place.
Other points mentioned were: the system's immediate response ("you put it together, and it works"
(P07)), and the possibility to perform quick cycles of trial-and-error; the system's versatility ("you
can build different shapes, [...] connect the blocks in various orientations, [...] explore
combinations" (P05)).
Finally, three participants said that the system could be used as a tool to help to have ideas (P01,
P06, P08), and two participants mentioned it could potentially work as a "creative trigger" (P01,
P03).
7.2.1.5.2. Negative Points
The most highlighted negative points were: (1) the current mapping and sound (60% of the
participants). One participant described his experience with the current mapping as "slightly
disappointing" (P07), while another commented on the limited bank of sound and controls: "As
far as sonic exploration, I reached the end very fast" (P01); (2) its ergonomic features (mentioned
by 50% of the participants). One participant said: "It is big enough to pick up and move around,
but small enough that you do not feel you can drop it" (P01). Another participant mentioned that
the cubic shape does not invite one to touch it "because it is uncomfortable" (P09), a fact
mentioned by four other participants; and (3) the fragility of the current prototype (50% of the
participants). Five participants felt that the system was not robust, and, amongst them, two
mentioned they were afraid of breaking it somehow (though this never happened in the trials).
One participant mentioned: "I think that in performance, my feeling is that there are already
interfaces with which you can interact better" (P10).
86
7.2.1.5.3. Suggestions
Three participants mentioned that even though the system presents various controllers, "all you
have is two hands" (P06), and, one participant suggested richer combinations of control on a
single block by attaching different sensors to it. Two other participants suggested the addition of
a sequencer or a looper, because, as one of the participants said, "you don't need to have hand
busy, you [can] build [musical] layers" (P02). Another participant mentioned: "you can let the
system run instead of being in charge all the time, actually making a gesture for creating the
sound" (P10).
7.2.1.5.4. Contexts of Use
When asked about possible contexts of use in future version, the participants mentioned:
"brainstorming tool" (P01), "creative trigger" (P03), "creativity support tool" (P07), "quick prototype
physical design ideas" (P01), "mapping testbed" (P05), "preliminary test phase" (P05), "explore
ideas" (P08), "generate ideas" (P01), "experiment mappings" (P03). All of these answers are
encouraging since they denote how the toolkit and its associated methodology promoted the
generation and quick exploration of ideas to our participants, which was our main initial
hypothesis.
Besides, the toolkit was also interpreted as an instrument that could be used on performances,
for "artistic practice" (P05), as a "DJ controller" (P10), to "control effects" (P02). It was also related
to "versatile generic controller" (P08), for "video editing" (P05) or "light control panel" (P08).
Additionally, the participants commented about its use with children as a music learning tool or
as a musical game, at a classroom, or with people with disabilities during rehabilitation sessions.
7.2.1.6. Discussion
Although this first experiment is not a formal evaluation of our approach, the results indicate that
we were on a promising path. Participant responses concerning positive points and context of
use (e.g., "creative trigger", "brainstorming tool", "explore ideas", etc.) support our hypothesis that
Probatio provides structured paths for generating ideas through fast, functional prototyping ("you
put it, and it works", "reduce time to trial-and-error", "[the connection] is pretty solid"). There are
also confirmations that the proposed system could lower the entry barrier for designing or
customizing DMIs thanks to appropriate physical affordances ("pick me up and do something with
me", "quite intuitive", "self-explainable", etc.). Although we intended to provide only a toolkit for
prototyping DMI, some participants suggested that it could be used as the final DMI itself. We
believe that if the prototype were non-functional or did not properly react to input commands in
real time, this perception would not exist.
The ergonomic issues and the lack of robustness mentioned by the participants are directly
related to the straight-angled shapes and the material chosen for the current version of the
system. Following the approach of quicker cycles of prototyping, we designed this version of
Probatio to be the simplest possible concretization of our approach in order to validate its founding
87
hypotheses. We opted for simpler shapes and materials in detriment to ergonomics and
robustness only for the sake of speed. These are clearly issues that should be taken into account
for the next development cycles.
Finally, the negative mentions and participants' frustration about the current sound and mapping
produced by the prototype were expected due to our choice to exclude computer-based mapping
tools from this experiment in order to focus on the physical toolkit and design methodology.
Obviously, designing and evaluating an appropriate mapping interface for Probatio is mandatory
for the completeness of the approach, but it is a research problem per se that we are currently
working on.
88
8. EVALUATION OF PROBATIO 0.2 In this chapter, we present the version 0.2 of Probatio and an experiment comparing this version
with a generic sensor toolkit.
8.1. Evolution from Probatio 0.1 For Probatio 0.2 (see Figure 8.1), we analyzed the comments from the exploratory study and
implemented new features. As mentioned before, the most common negative point raised by the
participants regards the mapping and the sound. Participants mentioned that the sound result
was disappointing and the total combination possibilities were easily reachable. Mainly due to its
cubic shapes and the repealing edges, another weakness pointed by the participants concerns
the ergonomics of the supports and blocks. As a suggestion, the participants mentioned that it
could be interesting to aggregate different sensors in one block.
Related to those issues, development actions taken for this second version focusses on four
aspects: (1) the increase of blocks quantity; (2) the possibility of changing the mapping strategy;
(3) the enhancement of sound output module; (4) the addition of curved shapes on components;
(X) the integration of multiple sensors in one block.
Additionally, due to observation during the first evaluation, we made other enhancements: (5)
modification on the way the arm supports are connected; (6) adaptation on the way the supports
are connected to the hub; (7) hub enclosure; (8) modifications to decrease the friction between
the blocks and the slots border.
Figure 8.1: Probatio 0.2: (a) hub, (b) three taps, (c) fretless, (d) limiters for the sides, (e) four-slots support with curved edges, (f) four-slots support, (g) locks for four slots support, (h) cradle with
connector on the side, (i) cradle with connector on the bottom, (j) three-by-three support, (k) breath, (l) bellows, (m) buttons, (n) one tap, (o) turntable, (p) knobs, (q) crank, (r) joystick.
89
8.1.1. Number of Blocks and Multiple Sensors
For this version, we built more eight blocks and modified the four buttons blocks to have two
buttons instead. The new blocks are:
• Breath: inspired by the mouthpiece of wind instruments. It is made of a wind sensor based
on temperature measurement, which is commercially available for drones. The user's breath
rises the temperature, and the value can be measured.
• Two buttons: still inspired by instruments that have keys, switches or valves. We built more
three blocks with this configuration in an attempt to increase the possible ways of selecting
or triggering discrete events.
• One tap: inspired by percussion instruments. It consists of a piezo element attached to the
upper face of the block. The user strikes the top of the block and the piezo generates a current
that can be measured. The dynamics of the strike interferes on the output value: the stronger
the strike, the higher the value.
• Three taps: works in a similar way as one tap. This block tries to incorporate a suggestion
collected in the exploratory evaluation which was to integrate into one block multiples
sensors. It is inspired by the tambourine skin.
• Knobs: inspired by the presence of knobs in instruments such as electric guitars,
synthesizers, etc. It has two potentiometers with knob head attached to them.
• Joystick: it is related to the integration of different sensors into one block. It is made of a
game controller spare part and comprises two small potentiometers attached to a metal stick.
The user moves horizontally; the horizontal potentiometer changes its value. Vertically, the
vertical potentiometer is activated. Its normal position is in the middle, thus, with no user
action, both vertical and horizontal potentiometers marks half of the value each.
8.1.2. Changing Mapping Strategy
Probatio 0.2 uses the libmapper ecosystem (MALLOCH; SINCLAIR; WANDERLEY, 2014).
Libmapper is a library that can be used in different software languages and allows an application
to define input and output signals that can be manipulated through the network and mapped to
various devices on-the-fly without recompiling the code, or resetting any system. By using
libmapper, Probatio can leverage existing sound synthesizers that are already built as part of the
libmapper environment and can be easily integrated with musical software such as Max and Pure
Data (via the library), and digital audio workstations (via MIDI).
Using a web browser, the user can change the mapping strategy by using a graphical user
interface developed for libmapper called Webmapper, which consists of a tabular interface on
which the user can connect gestural input parameters to sound output parameters.
90
8.1.3. Sound Output Module
The sound output was developed in Max 7, using the percolate library and a simple
implementation of a modular synthesizer developed with Max/MSP default objects.
Besides other features, percolate (TRUEMAN; DUBOIS, 2015) implements a series of STK
(Synthesis Toolkit) (COOK; SCAVONE, 2004) physical models of sound synthesis and allows the
manipulation of synthesis parameters through Max 7 interface.
The libmapper externals allow the integration of libmapper environment with Max. Therefore, the
parameter inputs of both Probatio and the sensors toolkit could be mapped into the output
parameters of the physical models in Max.
We chose the physical models of a mandolin and a flute. For the mandolin, the participant could
control the following parameters: pluck attack, detuning, body size, general volume, continuous
frequency, and discrete frequency inside a chosen musical scale. For the flute, the parameters
available were: breath pressure, tone hole state, register state, general volume, continuous
frequency, and discrete frequency inside a chosen musical scale.
Additionally, for a simple synthesizer, the available parameters were: ADSR envelope, an
oscillator with three types of waveforms (sine, sawtooth, square), resonance, cutoff frequency of
a lowpass filter, note trigger, general volume, continuous frequency, and discrete frequency inside
a chosen musical scale.
8.1.4. Curved Shapes
Attempting to avoid the use edges, we experimented with adding curved shapes in a new arm
support. We chose to experiment on an arm in order to allow the user to perform smooth hand
translation between different positions. We used a technique of digital fabrication that allowed to
bend the MDF by placing kerfs on the board. As a downside of this new configuration, the new
arm support lost the side connections.
8.1.5. Connection Arm Support
In the previous version, the arm support could just connect to the communication bus through the
4x4 support. This fact limited the users to use it standalone. For broadening the possibilities and
allowing this use, we introduced in this version two cradles in which any block (and therefore, the
base of the arm) can be placed.
8.1.6. Protection and Connections to the Hub
We improved the connection from the supports to the hub by replacing the jump wires headers
by RJ-12 connectors. This improvement gave more stability to the connection allowing the user
to move the components without the risk of damaging the connection.
91
Also to improve the connection and avoid loose contacts, the hub was inserted in a MDF box.
8.1.7. Friction of Blocks and Slots
The spring-loaded pins of the block's side surface have a ledge of less than a millimeter, which
made difficult the insertion in the slot. For solving this issue, we opened a two-millimeters cavity
on the slot's border that allows the block to pass freely without sticking.
We noticed that the MDF material suffers from a sensitive interference of humidity which causes
its dimensions to increase. Due to this fact, some block's insertion became difficult. To address
this issue, we sanded the block's surfaces that enter made contact to the slot's border.
Also, we observed that the fretless object, which was originally designed as a 2-unit block with
one block with a hot connection and other only as an additional support, was hard to be inserted
having due to misalignment of the blocks, which provoked friction. For that, we removed the
additional block, decreasing the contact area.
8.2. Experiment In this section, we present the second cycle of evaluation considering Probatio 0.2.
8.2.1. Objectives
We conducted a controlled experiment to analyze the effect of using Probatio: a) on the time
duration to achieve a DMI functional prototype, b) on the cycles of idea exploration and evaluation,
c) on the diversity of the explored possibilities, and d) on the user involvement with the system.
Following the Interaction Design approach (PREECE et al., 2015), in this experiment, we focused
on usability and user experience goals. For usability, we considered three principles:
effectiveness (the ability to accomplish results with quality), efficiency (used resources to achieve
the results), and satisfaction (user’s subjective reactions) (BROOKE, 1996). Regarding user
experience, we observed positive and negative aspects of the interaction, such as user’s
engagement and frustration with the system.
As Probatio is a prototyping toolkit for idea experimentation, we consider that a measure of
effectiveness is the number of cycles of idea generation and idea evaluation, and the diversity of
the exploration of possibilities. According to Camburn (2015a), Beaudouin-lafon (2000c), and
Von Hippel (2001), the quality of the outputs can be related to the number of these cycles,
because the user will be able to modify the prototype in order to achieve adequate results
(CAMBURN et al., 2015) (BEAUDOUIN-LAFON; MACKAY, 2000) (VON HIPPEL, 2001).
Efficiency can be evaluated taking into account the time the user needs to obtain a functional
prototype. Satisfaction is assessed by understanding the engagement, and frustration of the users
when using the system.
92
8.2.2. Design
We used a within-subjects design, i.e. all participants used Probatio, and a generic sensor
system. The task was to experiment musical interaction ideas by building and modifying functional
DMI prototypes.
According to Preece et al. (2015), besides reducing the interference of individual differences, the
advantage of the within-subjects design is that the sample of participants can be reduced to half
when comparing to other approaches such as between-users design, where different groups are
submitted to different conditions. However, the order of the conditions can lead to biased results
due to learning from the first condition (PREECE et al., 2015). To deal with that, counterbalancing
measures should be taken. For instance, one approach is to randomly choose half of the
participants to start with one condition and the other half to start with the other condition.
The independent variable was the use of a system, and it was evaluated with two levels:
• Condition X (Probatio) and
• Condition Y (a generic sensor toolkit).
As it will be described in details in section 8.2.6.3, the generic sensor toolkit presents a set of
sensors that can be connected to a microcontroller using jumper wires and breadboard. It will
function as a baseline to compare aspects such as the presence of physical structure, and way
of connecting items.
Our dependent variables (DV) are:
a) the time duration to achieve a functional prototype,
b) the number of cycles of idea exploration,
c) the number of distinct items used during the idea exploration, and
d) the overall user experience.
Variables (a), (b), and (c) are intrinsically quantitative, can be measured through observation.
However, we decided not to reduce user satisfaction to quantifiable values only. Considering
discussions on the HCI literature (LAW et al., 2009), we opted to use qualitative methods to
discuss the user experience.
Probatio is a tool for helping DMI designers to experiment ideas through a hands-on approach by
having immediate functional prototypes. It is expected that the cycles of idea exploration become
shorter and more numerous in times. Our hypothesis is that by using Probatio, the user achieves
prototypes in less time and performs diverse modifications or adaptations in these prototypes.
Considering the user experience, our hypothesis is that the user feels more engaged, and less
frustrated when using Probatio as compared to the generic sensor toolkit. We summarize our
hypotheses in Figure 8.2.
93
Figure 8.2: Summary of hypotheses
8.2.3. Methods for Data Collection
8.2.3.1. Video Recording
In order to measure the time and the number of experimentation cycles, we decided to record video and audio of the participants’ interactions with each system.
8.2.3.2. Questionnaire
To help us understand user experience, we decided to use a 5-points Likert-based scale
questionnaire, a comparative questionnaire, and a semi-structured interview.
8.2.3.2.1. System Usability Scale
The questionnaire is based on the System Usability Scale (SUS), which is intended to be a “quick
and dirty usability scale” (BROOKE, 1996). Bangor et al. (2008) present four reasons for using
SUS to assess the usability of a product: (1) the scale is not dependent on the technology used
in the system, (2) the set of questions is easily understandable by researchers and participants,
(3) the result is a single score number that makes comparison simpler, and (4) the set of question
and the scale is not restricted by property or trademarks (BANGOR; KORTUM; MILLER, 2008).
According to Lewis et al.’s (2009), for comparative within-subject experiments, a sample size of
at least 12 participants is recommended (LEWIS; SAURO, 2009).
The 10-questions SUS survey is:
1) I think that I would like to use this system frequently.
2) I found the system unnecessarily complex.
3) I thought the system was easy to use.
4) I think that I would need the support of a technical person to be able to use this system.
5) I found the various functions in this system were well integrated.
6) I thought there was too much inconsistency in this system.
7) I would imagine that most people would learn to use this system very quickly.
8) I found the system very cumbersome to use.
9) I felt very confident using the system.
10) I needed to learn a lot of things before I could get going with this system.
94
8.2.3.2.2. Exploratory Questions
Additionally, we formulated three exploratory questions to understand user’s impression about
how fast a resulting prototype can be achieved. Although there is a recommendation in literature
that Likert-based questions should be extensively tested before the main application to avoid
misinterpretation (BROOKE, 1996), due to constraints in time and participants’ numbers we
decided to present the question without a prior test. Therefore, we label these questions
exploratory, and we report the results with caution because of the chance of misinterpretation.
• I think that the system allowed me to experiment diverse ways of interaction
• I think that the system allowed me to explore in a fast way different interactions
• I think that the system showed me possibilities that I had not seen
8.2.3.3. Comparative Questions
Besides the SUS survey and the exploratory questions, we decided to present a comparative
questionnaire which comprises 18 fill-the-gap sentences whose multiple choice answers were:
“Both”, “Neither”, “Condition X”, and “Condition Y”. The sentences attempted to assess negative
and positive aspects of the experience (e.g. frustration, engagement, and perceived level of
difficulty). We decided to use a comparative approach to contrast the user’s impressions about
the system in an attempt to extract more information. The sentences were:
1) I think I achieved more interesting musical results using _____.
2) I felt I tested more musical interactions using _____.
3) I felt I could try more things using _____.
4) I would imagine that most people would learn to use _____ faster.
5) I think the process was more laborious when I used _____.
6) I think I achieved faster results using _____.
7) I felt more engaged using _____.
8) I felt more bored using _____.
9) I felt more frustrated using _____.
10) I felt more confident using _____.
11) I felt that _____ were the most challenging for me.
12) Comparing the two systems, I felt more creative using _____.
13) I felt that _____ were the most inspiring for me.
14) I think I've explored everything that _____ had to offer me.
15) I think I'd like to use _____ more often.
16) I think I understood the operation of _____ more.
17) I thought that _____ were the most complicated of the two systems.
18) I thought that _____ were the easiest to use.
95
8.2.3.4. Semi-structured Interview
To have an in-depth understanding of the participants’ experience and better comprehend the
impression they had of both systems, we performed a semi-structured interview that covered
topics such as strengths and weakness of each physical interface, challenge, frustration,
engagement, and impressions about musical interaction.
The semi-structured interview followed the topics below:
1) Describe the experience with each system. Please, explain.
2) Describe strengths and weakness of each system. Why?
3) If applicable, describe difficulties during each interaction. Why?
4) If applicable, describe moments that you felt frustrated. Why?
5) If applicable, describe moments that you felt engaged. Why?
6) If available, which interface would you use in the future? Why?
7) Which system provided a better way for exploring musical interactions? Why?
8) Which system provided a faster way for testing an idea? Why?
9) If you wish, give suggestions, comments, or improvements.
8.2.4. Methods for Quantitative Analysis
8.2.4.1. Video Analysis
To analyze the quantitative variables, we adopted video analysis based on instrumental
interaction analysis (JORDAN; HENDERSON, 1995). Instrumental interaction is a set of “activities
driven by the manipulation of physical objects” (XAMBÓ, 2015), this concept must not be
confused with the instrumental interaction proposed by Beaudouin-Lafon (2000), which is an
interaction model to describe the coupling instruments between physical world and on-screen
objects (BEAUDOUIN-LAFON, 2000). According to Jordan and Henderson (1995), the video is
beneficial over basic written annotation during observations because it provides ways of revisiting
sequences multiple times (JORDAN; HENDERSON, 1995).
To annotate, navigate, and visualize the recorded videos, we used ChronoViz, which is an open-
source tool for annotating and navigating through time-coded data (FOUSE et al., 2011).
Considering the concept of segments presented by Jordan and Henderson (1995), we analyzed
the data and associated codes (or categories) to sub clips of video in which the users presented
a similar intention during the interaction.
For instance, the user placing a block in the slot is coded as “Mounting”. For the definition of the
coding scheme, we followed a bottom-up approach: we watched all the videos, found recurrent
actions, marked the beginning and end time, assigned preliminary codes, revisited the data, and
adapted the codes.
96
8.2.4.2. System Usability Scale Score
To measure the SUS value of each system, we followed the procedure presented by (BROOKE,
1996), which aims to balance the positive and negative sentences, presenting an overall scale
from 0 to 100. This scale should not be confused with percentage (BROOKE, 2013).
Bangor et. al (2008) presents an anecdotal way of interpreting SUS values: products which SUS
scores above 70 are “passable”, between the high 70s and upper 80s are better and truly superior
products score better than 90. Products below 70 should be enhanced (BANGOR; KORTUM;
MILLER, 2008).
A recommendation by Brooke (1996) is that the individual answers of the SUS survey do not
represent a particular result, because of their correlation. Therefore, only the combined result
should be considered (BROOKE, 1996).
8.2.4.3. Paired-samples t-test
To determine whether there was a statistically significant difference between the SUS means of
condition X and condition Y, we used a paired-samples t-test. Paired t-test are used in within-
subjects design to reveal if the mean difference between paired observations is statistically
significantly different from zero (SHESKIN, 2003). To use this method, the assumption is that the
dependent variable is measured at the continuous level, and the independent variable comprises
two categorical related groups (in our case, the same participant tests two systems). Therefore,
we chose this method because it suits our experiments variables and within-subjects design.
This hypothesis test was also used to determine the significance of the mean difference of the
duration to build the prototype, as well as the number of cycles in the two conditions.
8.2.4.4. Wilcoxon Signed-Rank Test
For the three exploratory Likert-based questions, we decide to use to Wilcoxon Signed-Rank Test to determine whether there was a statistically significant median difference between the two
conditions. Wilcoxon Signed-Rank Test is a hypothesis test for within-subjects design which
considers ordinal data. It can be considered the nonparametric equivalent to the paired t-test
(SHESKIN, 2003). Instead of considering the mean difference between the paired observation,
the method considers the median difference.
Much has been discussed in HCI literature that one should or should not treat Liker-based
questions results as interval data (KAPTEIN; NASS; MARKOPOULOS, 2010). A possible bias,
for instance, is that the participants’ perception of the Likert-scale may differ: for one participant,
the hypothetical distance between “Strongly Agree” to “Agree” may differ from another participant.
This makes the continuous comparison unmatchable. Clason (1994) suggests that Likert-scale
results should be treated as ordinal value (CLASON; DORMODY, 1994). For that, the commonly
used t-test should be replaced by a corresponding nonparametric method. Therefore, we opted
to use Wilcoxon Signed-Rank Test with Likert-based questions.
97
8.2.5. Methods for Qualitative Analysis
8.2.5.1. Thematic Analysis
Thematic analysis is a commonly used method in qualitative research in which themes or patterns
are analyzed and identified within the data (BRAUN; CLARKE, 2006). The themes groups similar
responses, or impressions across different sources of data. We used a top-down approach to
identify recurring themes and topics from the interviews’ transcripts. Besides, we determined the
prevalence of themes using the number of different participants whose quotations were related
to the themes. Instead of identifying the themes at a latent or interpretative level, we adopted the
semantic or explicit level of interpretation, which considers what was said by the participants, and
not what could be beyond what the participant said or what is written.
8.2.5.2. Coding Methods
According to Saldaña (2009), a code is “a word or short phrase that symbolically assigns a
summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based
or visual data” (SALDAÑA, 2009). Coding is the basic activity to perform a thematic analysis, and
we used the following coding methods presented by Saldaña (2009): Structural Coding,
Evaluation Coding, and Emotion Coding, that will be discussed in details in the Qualitative Results
section.
8.2.6. Setup
The experiment setup comprises three groups of materials: the software, the hardware, and the
experimental physical interfaces.
In an attempt to develop an environment of prototyping DMIs, we followed the DMI model
presented by Miranda and Wanderley (2006) (MIRANDA; WANDERLEY, 2006). The experiment
setup covered gestural control input, with the experimental physical interfaces; mapping, with
libmapper and its GUI (called Webmapper); and sound output module, with synthesizers
developed in Max/MSP.
8.2.6.1. Hardware
The software ran on an Intel Core i7 MacBook Pro Retina. A 24” Dell U2413 LCD Monitor with a
1920x1200 pixel resolution was used as the primary display, and the screen of the MacBook used
as the secondary one. On the primary display, right in front of the participant, the windows of
Webmapper and Max were placed, and, on the secondary display, the window of the plotter was
positioned. The speaker system was composed of a subwoofer and two satellites speakers. A
wireless Magic Mouse 2 was made available for the participant to interact with the GUI.
98
For registering the experiment, a DSLR Camera Canon Rebel T4 with an 18-55mm lens was
placed on the left side of the participant in a tripod, far enough for not interfering in the experiment.
Figure 8.3 and Figure 8.4 illustrate the desk layout for Probatio and GSToolkit, respectively.
Figure 8.3: Desk layout for Probatio: (a) Probatio, (b) LCD monitor, the main screen, (c) MacBook
Pro, the second screen, (d) Magic Mouse 2, (e) chair for the participant, (f) camera
Figure 8.4: Desk layout for GSToolkit: (a) GSToolkit, (b) LCD monitor, the main screen, (c) MacBook Pro, the second screen, (d) Magic Mouse 2, (e) chair for the participant, (f) camera.
8.2.6.2. Software: Mapping and Sound
We adapted the software in Java which is responsible for the communication between Probatio
and the computer via serial port to work also with GSToolkit. As mentioned before, the software
is also accountable for plotting the input values of on the screen.
Kept the same for both experiment conditions, the Webmapper was used as the mapping
graphical user interface.
Webmapper GUI and the Max patches in Presentation Mode were placed on the main screen.
The window with sensor values plots, developed in Processing, was located in the secondary
screen on the left-hand side of the participant. Figure 8.5 presents the main screen configuration,
and Figure 8.6 shows the window with plotted input values of the system (in this example,
Probatio’s items).
99
Figure 8.5: Graphical user interface on the main screen
Figure 8.6: Secondary screen with the plotter values of Probatio's items
For performing the mapping in Webmapper, the participant had to drag the mouse pointer from
the input-parameter located on the left-hand side and release it over the output parameter located
on the right-hand side. To delete a mapping connection, the participant had to click on the curve
which represented the connection and press the Delete key.
8.2.6.3. Generic Sensor Toolkit
The generic sensor toolkit (or GSToolkit to simplify for further description) is a system that we
developed exclusively for this experiment (Figure 8.7).
100
Figure 8.7: GSToolkit's components: 8 buttons, 4 piezos, 1 membrane potentiometer, 1 force-sensing resistor, 1 joystick, 2 rotary encoders, 2 rotary potentiometers, 1 hall-effect sensor, 1
breath sensor
The toolkit consists of 21 sensors, an Arduino Mega 2560, a 400-tie-points clear solderless
breadboard, and a set of A5-size cards with instructions (Figure 8.8) on how to assemble the
circuit of the sensors.
The sensors were presented with their circuit pre-assembled. In other words, all the needed
electronic components (such as resistors) were bundled together with the sensor and sealed with
hot glue. Our objective was to succinctly enclose technical details that would require for the
participants to have skills related to identifying electronic components and correctly using them.
This approach is similar to the iCubeX, a sensor toolkit developed by Mulder (1995) (MULDER,
1995). We believe that this complexity layer would reduce the ability to perform a comparative
experiment between Probatio and GSToolkit.
Figure 8.8: GSToolkit's instructions cards
101
Each sensor had a correspondent instruction card which presented the wiring connections
between the sensors, the breadboard, and the Arduino board. We used this approach as a
simplification of the internet search for information regarding a specific sensor. Our main objective
was to create a controlled environment with a reduced complexity regarding outside variables
such as internet speed, individual strategy to search such information online, previous knowledge
about existing source of such information, or other factors that would interfere in the session
duration.
Furthermore, we pre-programmed the Arduino microcontroller, so the participant did not have to
deal with coding or to use Arduino IDE. We chose to use gate pins that would activate or
deactivate the reading of a specific analog or digital port of the Arduino. Each sensor had its own
associated gate pin. Instantaneously, after following the instructions card, the sensor values were
plotted in a graphical user interface on the secondary display.
8.2.6.4. Correspondence Between Systems
Probatio presented all the correspondent sensors used in GSToolkit, but Probatio’s items
encapsulated more than one item in one block, besides having the physical structure. Figure 8.9
presents the correspondence between the two systems.
102
Figure 8.9: Correspondence between GSToolkit's items and Probatio's items
The structure and encapsulation cause perceptible differences in the interaction with each
system. For example, the block Two Buttons is equivalent to using two buttons in GSToolkit at
once, i.e. in Probatio with just one connection the user obtains two degrees of freedom (DoF),
and in GSToolkit, to obtain the same number of DoFs the user has to perform two assembly
actions. The block Fretless combines two sensors of GSToolkit (membrane potentiometer and
force-sensing resistor) in just one surface. Although using a Hall Effect Sensor inside, the block
Bellows is mechanically constrained by the upper part movement, while the user in GSToolkit can
experiment different uses with the sensor. The blocks Turntable and Crank present elements (the
disc and the crank) that change the interaction with the Rotary Encoder.
Although we maintain a sensor type equivalence in both systems, because of the structure and
the encapsulation we obtain different possible uses. In sum, in this experiment, we attempt to
assess the impact of these differences in the overall prototyping process, and in the user’s
engagement with the system.
103
8.2.7. Technical Test Pilot
To test the system setup with Probatio and GSToolkit in an attempt to discover some technical
issues that could interfere with further experiments results, we organized a workshop with the
students of the Computer Music The discipline presented a mixed group of programmers and
musicians from Computer Science, Computer Engineering, and Music courses.
For all the participants, we briefly explained how Webmapper and Max objects work. Then, the
class was divided taking into account programming skills, and musical skills and two mixed groups
were formed, group A and B, both with 9 participants. Group A received the GSToolkit system,
and group B used Probatio. The groups were then sent to different rooms, and they freely
experimented the systems for one hour and a half. They were asked to experiment musical
interactions focusing on finding ways to control the parameters of sound objects in Max/MSP. The
sessions were recorded on video for later analysis. Figure 8.10 shows the two groups
experimenting the systems.
Figure 8.10: Group A (GSToolkit) and Group B (Probatio)
The highlights of this preliminary test were:
• Probatio presented several errors mainly due to loose contacts between the spring-loaded
pins and the slots connectors. It caused the software to crash, which demanded the system
to be reset more than 5 times. The participants demonstrated frustration by sentences such
as or “the idea is exciting, but there is a lot of errors”. Besides, they were limiting their
interaction to avoid causing the system to crash, as illustrated by the sentence “It is better to
keep it there [talking about a block] to avoid causing errors”.
• The sensor kit presented no major errors.
• The group B dynamics can be described as a person in the middle doing all the work and the
other members around giving suggestions and instructions. The same person who was
mounting the sensors make the connections on the mapping GUI.
• Group A more people participated in the hands-on activity. More than five participants
grabbed Probatio’s blocks and inserted them in the slots. Only one person though controlled
the mapping GUI.
This pilot showed:
104
• The vulnerability of Probatio’s connections in two aspects: (1) the loose contacts between the
spring-loaded pins and the support’s connectors, and (2) the jumper wires connections
between Probatio’s supports and the hub.
• The lack of fault tolerance and system recovery in Probatio setup after the occurrence of a
bad connection. Several times, the system fully reset after simple errors.
• The stability of the sensor toolkit
For the main experiment, we reviewed Probatio structure and connections. Regarding the fault
tolerance issue, as an urgent solution, we used a second Arduino (Uno version). It works like a
facade: it is always connected to the computer even if the Arduino Mega resets due to Probatio
errors. It is not the most elegant solution, but we managed to deal with the errors in the main
experiment with this buffer-like bypass.
8.2.8. Participants
For the main experiment, we selected our participants based on a list produced in the early
exploration phase (Batebit project (BARBOSA et al., 2015a)) of popular musicians and musical
producers from Recife, Brazil, with interest or experience in music technology.
We enrolled 19 volunteers ranging in age from 19 to 50 years old (mean age 34.74, SD 8.87), 17
males and 2 females. All of the participants had experience in digital technology in music, played
at least one musical instrument, and had little or no experience with microcontrollers, digital
electronics, sensors, and programming languages (Figure 8.11).
Figure 8.11: Participants' profile. List of covered topics in order of mentioned experience: Musical
Experience, Digital Audio Workstations, Digital Instruments, Sound Synthesis, MIDI Keyboard,
105
Electric Instruments, Acoustic Instruments, Musical Theory, Mapping, Electronic Instruments, MIDI Controller Non−keyboard, Digital Electronics, Physical Interfaces Music, Artistic Programming
Language, Microcontrollers, Self−made MIDI, General Programming Language
Five participants were not able to perform the complete session due to schedule unavailability,
so we reduced the duration of their experiment sessions. Instead of not considering all the results
from these participants, we divided the 19 participants into two groups (G05 and G14) (Figure
8.12). For G14, we performed quantitative and qualitative analysis, and, for G05, we analyzed
only qualitatively the transcript of their interviews. Figure 8.13 and Figure 8.14 present the
participants using Probatio and GSToolkit respectively.
Figure 8.12: Group division. The participants’ codes follow the order of experiment sessions.
Figure 8.13: 19 participants using GSToolkit
106
Figure 8.14: 19 participants using Probatio
8.2.9. Experiment Protocol
The sessions took place with one participant at the time and lasted approximately 100 minutes
each (some adaptations were made for the group of five participants that could not spend long
durations, G05). As a counterbalancing measure, we randomly defined the order of the conditions
X (Probatio) and Y (GSToolkit) prior to the beginning of the experiment.
The participants were warned that the experiment would cause no harm and by continuing in the
session, they agreed of allowing the use of their image for research analysis purpose.
Each session followed the subsequent steps (illustrated in the Figure 8.15):
Figure 8.15: Within-subjects design with group X->Y starting with Probatio (condition X) followed by generic sensors (condition Y). And, group YX, using the generic sensors followed by Probatio
1. Brief Introduction (2 minutes): the researcher introduces the participant to the research,
explaining that the experiment is a part of a Ph.D. project which investigates new interfaces
for musical expression. The researcher states that the objective of the experiment is not to
107
measure individual abilities with digital technology and music but to understand the use and
the experience of using two different physical interfaces for musical interaction.
2. Profiling questionnaire (5 minutes): the participant answers a questionnaire that covers
experience in music, background in musical instruments, digital electronics, and
programming languages.
3. Explanation about the GUI (5 minutes): the researcher presents the graphical user
interface to the participant and gives some examples of how the mapping from input
parameters can be made to the sound output parameters using the Webmapper. The
researcher also presents the three sound devices developed in Max and mentions their
parameters. For the didactic purpose and to be used in this initial explanation about the GUI,
two Max GUI-based input devices were developed: a low-frequency oscillator with six waves
in different frequencies generating six input parameters for mapping; and a graphical
interface with three buttons and three sliders.
4. Interaction with GUI (5 minutes): the participant freely explores the GUI by her/his own.
5. Brief explanation about System [X or Y] (5 minutes): the researcher places the first
system on the table at which the participant is sitting. The researcher mentions all parts by
name and briefly describes what each one does. The researcher mounts one part and
presents how the sensor graph will plot the value. Then, the researcher shows how the
parameters will appear on Webmapper. Finally, the researcher says that the participant can
freely explore and use the system to experiment musical interactions.
6. Interaction with System [X or Y] (25 minutes): the participant freely explores the physical
interface and the mapping GUI. The video and audio of this interaction session are recorded
for later analysis.
7. Questionnaire considering System [X or Y] (10 minutes): the participant is asked to
answer a questionnaire on the computer in front of her/him. In the first question of the
questionnaire, she/he identifies which system she/he firstly used. The questions follow the
Likert scale from 0 to 5 (strongly disagree - strongly agree). While the participant answers
the questions, the researcher removes the physical interface from the table.
8. Brief explanation about System [Y or X] (5 minutes): step 6 is repeated for the second
system.
9. Interaction with System [Y or X] (25 minutes): the participant freely explores the second
physical interface using the same mapping GUI. The video and audio of this interaction
session are recorded for later analysis.
10. Questionnaire considering System [Y or X] (10 minutes): the participant answers the
same questionnaire from step 8, identifying the second system in the beginning question.
11. Comparative questionnaire (5 minutes): the researcher presents a comparative
questionnaire to the participant. The questionnaire is similar to the previous set of questions,
108
but instead of Likert-scale answers, the questions have only four options: “Both”, “Neither”,
“System X”, “System Y”.
12. Semi-structured interview (10-15 minutes): the researcher conducts a semi-structured
interview with the participant covering points such as strengths and weakness of each
system, engagement, frustration and perceived challenge and time spent, difficulties,
integration with her/his own musical tools and suggestions for enhancement. The video and
audio of the interview are recorded for later analysis.
8.3. Results
8.3.1. Quantitative Analysis
8.3.1.1. Video Analysis
After cycles of analyzing the video data and assigning preliminary descriptions to video segments,
we defined the following list of codes, which represent the interaction phases:
8.3.1.1.1. Code Scheme
• Mounting: the participant arranges the physical setup using the system’s components. For
example, placing a block in a slot using Probatio or connecting sensor wires using elements
of GSToolkit.
• Mapping: the participant changes the focus from the physical interface to the computer
screen, grabs the mouse and uses the Webmapper interface to map inputs to outputs
parameters, or adjusts some elements such as musical scale in Max.
• Testing: the participant uses the elements of the physical interfaces and expects either visual
feedback from the sensor value plots or sound feedback from the sound output modules.
• Thinking: the participant does not perform any action either using the physical interface or
the graphical interface.
• Asking: the participant stops what she/he is doing to ask the researcher about some doubt
of the physical interface or GUI.
• Bug: the system stops working due to an error and the researcher intervenes to solve the
problem.
8.3.1.1.2. Duration of the Interaction Phases
In order to help to visualize the data, we plotted the video segments of each session as bar
graphs, presented in Figure 8.17 (legends placed separately in Figure 8.16).
109
Figure 8.16: Legend for the video segments
Figure 8.17: Time segments of the interaction phases
As can be seen, two sessions took much less than 25 minutes: P09_X and P19_Y. It happened
because the participants said they had tested everything they wanted and asked to stop.
Figure 8.18 presents only the Mounting phases bars of Figure 8.17, and it shows that the larger
area of purple-colored bars in condition Y indicates that GSToolkit demands more time for the
user to build a functional prototype when compared to Probatio as expected. This preliminary
visual analysis is supported by paired t-test performed to determine the statistically significant
mean difference between the duration of Mounting phases in both conditions (t = -8.5796, df =
13, p = 1,03E-06). It appears that participants take longer to build a functional prototype when
using GSToolkit (M = 675.14, SD = 235.17) as opposed to using Probatio (M = 150.14, SD =
64.18). The mean increase is 525 seconds, 95% Confidence Interval [392.8031, 657.1969].
Figure DUR05 shows that the mean duration of Mounting phases in condition Y is slightly over
four times the mean duration of the same phase in condition X.
110
Figure 8.18: Mounting phases are highlighted, and other phases are grayed out
No statistical significant difference was identified in Bug phases duration between Probatio and
GSToolkit (t = 0.57981, df = 13, p = 0.572). However, Figure 8.19 shows that the incidence of
bugs was larger in Probatio sessions than GSToolkit sessions (six times more errors occurred in
Probatio than in GSToolkit).
Figure 8.19: Bug phases are highlighted, and other phases are grayed out
To have a better understanding of these errors, we revisited the videos and gathered more details
(Figure 8.20). As can be seen, the majority of bugs is related to loose contacts between the blocks
111
and the connection slots. The errors that happened in GSToolkit seems to be related to user
misinterpreting the instruction cards and placing the wires in wrong position, and a sudden serial
port failure in the computer. Therefore, the errors are not directly related to the system
malfunctioning. Additionally, the longer duration of a bug segment is in session P06_Y. The error
took longer to be discovered by the researcher because of the number of wires presented in the
participant’s layout. P06 developed a strategy of mounting all the sensors he wanted first and
then testing them.
Figure 8.20: List of Bugs
Figure 8.21 shows the visual differences between the duration of the two conditions. Paired-
sample t-test revealed a statistically significant mean increase of 183.71 seconds, 95%
Confidence Interval [47.01276, 320.41581], (t = 2.9033, df = 13, p-value = 0.01233) in Mapping
phases between X and Y. Besides, there was also a significant mean increase of 269 seconds
95% Confidence Interval [156.3708, 381.6292] (t = 5.1598, df = 13, p-value = 0.0001835) in mean
duration of Testing phases of Probatio when compared with GSToolkit.
Figure 8.21: Mapping and Testing phases are highlighted, and other phases are grayed out
112
The analysis of the following phases did not present any significant difference according to
performed paired t-tests: Asking (t = 1.4488, df = 13, p-value = 0.1711), Bug (t = 0.57981, df =
13, p-value = 0.572), and Planning (t = 0.63864, df = 13, p-value = 0.5341).
Figure 8.22 displays the mean of interaction phase durations in seconds. As revealed by the
graph, the Mounting phase duration is highly affected by the condition X and Y. It was somehow
expected due to the simplicity of Probatio’s way of connecting the components in the support that
did not require either any instructions to be followed nor the slightly elevated level of attention to
assembling the circuits.
Figure 8.22: Mean of Duration in Seconds
113
8.3.1.1.3. Quantity of Transitions Between Phases
In order to assess the impact of the conditions X and Y in the quantity of cycles of idea generation
and idea evaluation, we computed the transitions between the interaction phases. To help the
visualization, we plotted a directed cyclic graph considering the total numbers of transitions
between the phases (Figure 8.23). The nodes represent the interaction phases, and the edges
represent the transitions. The color scale of the edges is based on a progression from a cold color
(blue) to a hot color (red). Also, the thickness of the edges is related to the number of transitions.
The flow of the edge follows a clockwise direction between the nodes, for example, considering
the edges between Testing and Mapping, the edge of the right represents the transitions from
Testing phase to Mapping phase, and the edge of left represents the transitions from Mapping to
Testing.
Figure 8.23: Transitions between phases. The origin-destination flow of the edges follows a
clockwise direction. The thickness represents the number of the transitions.
As the graph displays, there is a visual difference (in color and in thickness) in the Mapping-
Testing and Testing-Mapping transition between the two conditions. This is supported by the
following paired t-tests that determine the statistically significant mean difference between the
conditions (Figure 8.24):
• Mapping-Testing (t = 6.6458, df = 13, p-value = 1.60E-05), mean difference: 26.57143, 95%
confidence interval [17.93377, 35.20909].
• Testing-Mapping (t = 5.1988, df = 13, p-value = 0.0001715), mean difference: 22.42857, 95%
confidence interval [13.10833, 31.74881].
114
Figure 8.24: Mean of Transition's Number
As presented in the Figure 8.25, the number of Mapping-Testing, as well as Testing-Mapping,
transitions in X is slightly over twice as many transitions of the same kind in Y. Again, we can
argue that the reduced duration of Mounting phase might have influenced the increase of
Mapping-Testing and Testing-Mapping transitions. As the time and effort are not being consumed
in the Mounting phase, we may speculate that the participants able to perform more tests with
what they have.
Based on the data of this experiment, the use of Probatio seemed to contribute to the increase in
the number of cycles between interactions. If we consider the relevant correlation between the
interaction phases and the idea exploration-evaluation cycles, we can assume that the use of
Probatio facilitates in the increase of cycles exploration-evaluation.
Figure 8.25: Number of transitions between interaction phases
115
8.3.1.1.4. Quantity of Distinct Items Used
Additionally, to the dependent variables described before, we also tested a different variable that
can be related to the diversity of the exploration in the restricted design space of both systems.
This variable is the number of distinct items used in the sessions. Items for Probatio are the blocks
and for GSToolkit are the sensors. If the participant used one item several times, it was counted
as one.
A paired t-test revealed a statistically significant difference between the mean of numbers of
distinct items used in X and in Y (t = 4.6186, df = 13, p-value = 0.0004811). The mean difference
is 2.714286, 95% confidence interval [1.444678, 3.983893]. It means that the participants used
on average almost three items more in Probatio than in GSToolkit. We can speculate that the
ease of connection contributed to the expansion of exploration. Figure 8.26 illustrates the mean
difference between the two conditions.
Figure 8.26: Number of Distinct Items
8.3.1.1.5. Postures and ways of holding
As a secondary measure of diversity, we also analyzed the number of postures and ways of
holding the items of the systems. The tabletop position, where the items rest on the table, was
the only one observed in GSToolkit. Beyond the tabletop position, participants using Probatio
experimented different ways of placing blocks and holding the supports. The different positions
are described in the list below with the number of appearances and presented in Figure 8.27.
• Position A: (1x) Block [bellows] placed on the side of support [3x3]
• Position B: (1x) Block [breath] placed on the side of support [3x3]
116
• Position C: (1x) Block [knobs] placed on the side of support [3x3]
• Position D: (1x) Block [threetaps] placed on the side of support [3x3]
• Position E: (1x) Support [four-slots with curved edges] placed on the side of support [3x3]
• Position F: (1x) Support [four-slots] placed in support [3x3] as tower
• Position G: (2x) Support [four-slots with curved edges] placed in support [3x3] as tower
• Position H: (1x) Support [four-slots with curved edges] placed in support [cradle] and held
as clarinet
• Position I: (3x) Support [four-slots with curved edges] placed on the side of support [3x3] and held as guitar
Figure 8.27: Positions different than tabletop experimented by the participants
We may conclude that in Probatio the participants expanded the structural exploration, while in
GSToolkit they maintain the same tabletop position. The lack of structure in GSToolkit seems to
be the cause of this limited exploration. Since the participants had to assemble the circuit in a
reduced space, they had little options to enlarge their searching space.
8.3.1.2. Questionnaires
8.3.1.2.1. Comparing the System Usability Scale
We followed the method for calculating the System Usability Scale (BROOKE, 1996) for each
session was computed: ((Odd questions - 1)+(5 - Even questions))*2.5. Table SUS01 shows the
computed values.
A paired t-test showed a significant mean difference in the SUS values between conditions X and
Y (t = 2.5117, df = 13, p-value = 0.02601). In Figure 8.28, the means are presented with 95%
confidence interval, calculated according to (MOREY, 2008), which takes into account the within-
subjects experiment design. The mean difference is 13.03571 with a 95% confidence interval of
[1.823559, 24.247870].
117
Figure 8.28: Means of SUS Score
With a SUS score of 75.53, 95% CI [83.46, 67.61], according to (BANGOR; KORTUM; MILLER,
2008), Probatio is considered to be above average, which is 70. If we consider the lowest value
of the confidence interval, Probatio is slightly below average. Oppositely, GSToolkit scored 62.5,
95% CI [70.43, 54.57], and, therefore it would be considered below the average and candidate
for modifications. In sum, if we consider that SUS measures the subjective opinion of the usability
of a given system, and this opinion leads to the willing of using this system or not, Probatio would
be ranked in a better position than GSToolkit.
8.3.1.2.2. Explorative Questions
Due to the reduced sample size, we used the Exact Wilcoxon Signed-Rank Test following the
recommendation by Sheskin (2003) (SHESKIN, 2003). Exact calculation for large samples can
demand a costly computation, but for small samples, it is a more accurate method. The tests for
the three questions are presented below:
• Q18: Z = 1.889, p = 0.125
• Q19: Z = 1.354, p = 0.242
• Q20: Z = 2.111, p = 0.063
Thus, neither of the median differences of the three questions reached statistical significance. We
retain the null hypothesis that their median difference is equal to zero. Because of this result, we
can assume that the questions do not provide room for a comparative discussion. However, for
118
general reference only, the results are presented in Figure 8.29. As can be seen, in general, the
participants tended to evaluate the three questions with high values.
Figure 8.29: Results of Exploratory Questions
119
8.3.1.2.3. Comparative Questions
Figure 8.30 shows the percentages of answers for the six first questions of the comparative
survey. As can be seen, the majority of the participants chose Probatio for issues such as
interesting musical results, more musical interactions, and faster results. When asked about the
most laborious experience, almost 90% of the participants chose the GSToolkit. We can probably
state that, in terms of user’s perceptions, Probatio seems to be more associated to providing
quick ways to achieve more results.
Figure 8.30: Comparative questions: "01 - I think I achieved more interesting musical results using _____.", "02 - I think I achieved faster results using _____.", "03 - I felt more engaged using _____.",
"04 - I think I understood more the operation of _____.", "0
120
Figure 8.31 displays questions related to participant’s feelings, e.g. engagement and frustration
with the systems. As it is shown, the engagement and inspiration sensation measured by question
07 and 13, respectively, presents an almost equal distribution, with the highest percentage of both
systems. With that, we may conclude that both systems are equally engaging and inspiring for
the participants in this experiment. For the inspiration, the participants were more inclined to
Probatio.
Additionally, the majority of the participants responded that neither one of the systems was boring
and that GSToolkit was the most challenging of the two systems. We may summarize that even
though the participants found GSToolkit laborious, they engaged with the system. However,
almost one-third of the participants said that GSToolkit made them fell more frustrated. Frustration
is a feeling associated with not achieving what one desires. Therefore, we could say that although
GSToolkit was engaging and not boring, part of the participants did not accomplish the intended
results. Question 12 plot (Figure 8.31) shows that the user’s perception of creativity tends to be
greater using Probatio, but we cannot conclude that GSToolkit did not provide such feeling
because the majority of the responses was that both equally contributed to this feeling.
Figure 8.31: Comparative Questions: "07 - I felt more engaged using _____.", "08 - I felt more bored
using _____.", "09 - I felt more frustrated using _____.", "10 - I felt more confident using _____.", "11 - I felt that _____ were the most challenging for me.", "1 2 - Comparing the two systems, I felt
more creative using _____.", "13 - I felt that _____ were the most inspiring for me."
121
According to Figure 8.32, with both systems, the majority of the participants felt they did not reach
the level of total exploration. This extra room for exploration may contribute to the interpretation
that there is more potential for future engagement in both systems. Answers to questions 17 and
18 show that participants tend to say that the most complicated system is GSToolkit and the
easiest to use is Probatio. Considering the will of using the system more often, Probatio is better
ranked with almost half of the participant’s answers. These answers may contribute to the
conclusion that, due to the ease of use, quicker and faster ways of achieving results, Probatio is
a better candidate for future explorative incursions.
Figure 8.32: Comparative Questions: "14 - I think I've explored everything that _____ had to offer me.", "15 - I think I'd like to use _____ more often.", "16 - I think I understood the operation of _____ more.", "17 - I thought that _____ were the most complicated of the two systems.", "18 - I thought that _____ were the easiest to use."
Although the comparative questions may have limitations due to participants possibly
misinterpreting details, Probatio seems to be more associated with a quicker way of achieving
results and achieving more results.
122
8.3.1.3. Summary
In this Section, we present the summary of the quantitative analysis.
The results are:
• Mounting phases in GSToolkit were longer than in Probatio
• Bugs were more numerous in Probatio than in GSToolkit
• Testing and Mapping phases were longer in Probatio than in GSToolkit
• Mapping-Testing and Mapping-Testing transitions were more numerous in Probatio than in
GSToolkit
• The number of distinct items used in Probatio was greater than in GSToolkit
• System Usability Scale score of Probatio was above the considered average of 70
• SUS score of GSToolkit was below the considered average of 70
• SUS Score of Probatio was greater than GSToolkit’s
• Probatio seemed to be more associated with providing quick results
• Both systems seemed to be equally related to engagement and inspiration
• Neither system was considered boring
• The sensation of creativity tended to be greater using Probatio
The majority of the results corroborated with our expectative. However, the number of errors in
Probatio did not happen as projected. Surprisingly, participants using GSToolkit did not make as
many mistakes as we expected due to the harder connections.
8.3.2. Qualitative Analysis
The interviews were performed in Portuguese, transcribed and also analyzed in Portuguese. After
that, in order to present the results, we translated the used quotations and codes to English.
Our thematic analysis was cyclic and iterative. We chose to start with a top-down approach, in
which we defined the concrete elements present in the experiment. We used the method of
Structural Coding (SALDAÑA, 2009), which follows an initial structure to label and index the data
in order to make the access quicker.
Our top-down structural codes were:
• BLOCKS: covers comments about Probatio.
• SENSORS: comprises comments about GSToolkit.
• GUI: is composed of quotations about the graphical user interface.
• SOUND: is related to comments about the sound output.
After identifying each quotation with these codes, we started to categorize them in positive, or
negative comments. This is based on the Evaluation Coding, which defines positive and negative
magnitude codes to data (SALDAÑA, 2009).
123
In this first cycle, we also defined the following codes in a bottom-up approach, which means they
appeared during the coding:
• COMPARISONS: covers the sentences or paragraphs that were related to explicit
comparisons between BLOCKS and SENSORS.
• GENERAL: comprises general statements about the participant’s musical practice or
experience.
• PERSONAL: is related to sentences about personal characteristics of the participant.
• EXPERIMENT: consists of impressions about the experiment session.
Additionally, we followed the Emotion Coding method, which attempts to find parts of the text
related to feelings or sensations. This method is appropriate for exploring “interpersonal and
interpersonal participant experiences and actions” (SALDAÑA, 2009). The following codes were
defined and are presented with the number of participants in parenthesis that mentions related
sentences to the code: engagement (14), frustration (8), challenging (3), creativity (3), fun (3),
feeling like teacher sparrow or mad scientist (2), flow (2), rewarding (2), boredom (1), curious (1),
possessiveness (1), worried (1). The elevated number of “engagement” and “frustration” mentions
is related to the questions of the interview. Suggestions and contexts of use were also codes
defined during the first cycle.
Codes are the basic element of the thematic analysis (BRAUN; CLARKE, 2006). The theme is a
more abstract construction, and it is the result of the integration of related codes. For that, we
performed a second major cycle to identify common themes. The identified themes were:
8.3.2.1. Number of Components
Although one participant positively mentioned the number of blocks in Probatio as a pleasant environment to experiment ("It's a cool feeling, and since it has a lot of stuff, it's a really fun
feeling." (P12: 2)), two participants found that was too much information to process at first
("Because the blocks were a lot of things, a lot of information, a lot of possibilities" (P10:24)). One
of them also mentioned that the experience was overwhelming ("But I felt intimidated by a
number of things. I was a bit overwhelmed." (P15:3)).
8.3.2.2. Thinking Before Building and Experimenting Fewer Options
Three participants (P10, P15, P19) comment that GSToolkit process for building the prototype
induce them to follow a logic ("[The sensors] compel you to a logic." (P10:25)), and, because it
was laborious to connect the circuit ("The work required to assemble the sensors is much larger
than in the blocks, of course." (P19:6)), they took a strategy of thinking what they want before
and then building the prototype ("I think I thought a bit before in what I was going to do, as
opposed to fitting things." (P15:5) and "Then the process was, instead of trying something, I
thought first what I wanted" (P19:4)).
124
Because of that, some participants also mentioned they felt more focused ("There I was much
more focused" (P15:6) and "I felt more compelled to already have a focus than in a scattered
way" (P19:6)). Due to this focus, one of them stated that he experimented fewer options
("Instead of trying various things, I was more directed." (P19:3)).
In the opposite direction, Probatio might have provided an expansion in the experimentation space, as one participant said he could experiment more things at the same time ("I think one
took me a lot more to experiment with several things at the same time [the blocks], while the other
[the sensors] was more closed." (P19:7)).
Other participant felt that he experimented more possibilities, which caused him to feel more excited, and more engaged ("I think that, with the blocks, you can try more, so it gets you more
excited. It makes you more engaged in using that device." (P11:9)).
8.3.2.3. Following Instructions vs. Free Exploration
Regarding GSToolkit, the process of reading the instruction cards was mentioned as
challenging and rewarding ("Seeing the drawing and connect, it gives a stimulus .... 'Damn it,
I'm connecting a few wires here, and it's working.'" (P03:15)).
One participant felt that the cards help him attenuate the learning curve of using GSToolkit ("I
did not have a learning curve. I knew I had the cards each explaining what each sensor was and
how to connect them." (P15:8)). The participant suggests that with the cards, he did not have to
spend energy trying to understand the system, he just had to follow the instructions ("I had
to find out how those things were working" (P15:9)).
However, some participants mentioned the opposite. Even though Probatio did not have
instruction cards, in a comparative manner, one participant mentioned that with Probatio they did
not need to follow instructions ("You do not have to be looking at a manual to connect the stuff."
(P11:3)). Because, intuitively ("When I started interacting with the blocks system, I found them
quite intuitive." (P19:1)), they could mount a simple setup, and quickly achieve musical results.
8.3.2.4. Potential of the GSToolkit
In total, eight participants (P01, P04, P06, P07, P08, P09, P10, P14) made comments about the
potential of customization and more freedom to arrange the position of GSToolkit
components ("Sensors can give greater freedom because they are not already structured."
(P06:5)).
One participant felt they could arrange things the way he wanted ("It gave you the chance to
assemble the way you want." (P10:1)). Related to this, another participant mentioned that he
enjoyed more the experience with GSToolkit ("As I very much like to [...] adapt to my form, I
found the sensors more interesting" (P04:1)).
For some participants, the manipulation of wires ("I liked that feeling with the wire." (P03:9))
evoked a good sensation and feeling like a mad scientist ("Sense of openness to a world of
125
possibilities that you can develop with that there, a mad scientist thing" (P16:8)) or Gyro Gearloose (Disney’s fictional character, an inventor of Donald Duck’s universe. In Portuguese:
“Professor Pardal”) ("Here the sensors ended up being [...] I felt more like a Gyro Gearloose"
(P03:8)).
These positive impressions regarding GSToolkit’s structural freedom seem to go against the
negative positions about the lack of physical support and the restricted space of interaction
around the breadboard, mentioned by nine participants (P01, P02, P03, P08, P10, P11, P13, P14,
P16). This contradiction can be possibly explained by the fact that the participants seemed to be
talking about GSToolkit’s potential and not only about the implemented version for the
experiment (“The experience with sensors, it's more open, it gives you a sense of openness. It's
as if you knew, there was a very clear potential there" (P16:2)).
Comparing to this GSToolkit’s potential, Probatio seemed to be more limited ("It has a limit
because it already has pieces already pre-made. It has how to customize, but it cannot go on
such a deep level." (P10:6)).
8.3.2.5. Shape Limitations
One participant mentioned that Probatio was not ergonomic ("It is not very ergonomic." (P08:14))
and some reasons may have emerged, such as the cubic shapes and the pointy edges ("I think
the shapes, the very sharp edges, the square thing, is playful, but it needs to evolve a lot so I find
it comfortable to interact with really" (P16:24)).
Also concerning shape limitations of Probatio, one participant felt frustrated with the blocks that
did not fit together in Probatio ("Having a stuff that is bigger or smaller and does not fit right."
(P09:15)).
8.3.2.6. Physical Support
The presence of physical support in Probatio was positively highlighted by nine participants
(P01, P02, P03, P04, P05, P10, P11, P14, P15). For some participants, the object seemed to
cause an attraction ("It is here [the blocks] has a lust for the object, for the thing." (P08:31)).
Due to the structure, the Probatio’s blocks were related to musical instruments ("But I think the
blocks, they look more like an instrument" (P08:25)). As a consequence, one participant
mentioned that Probatio’s supports could awaken in the user a familiarity with instruments that already exist, and, even if the user does not play a certain instrument, one can connect with
known elements, be it shape or gesture. ("It has a lot of similarity to a wind instrument by the
format. So you kind of automatically make a reference, it connects with these canonical
languages, things that even people who do not have previous experience with musical
instruments can understand that there are languages there. I think this is cool"(P16:11)).
Because Probatio seemed to be closer to a final product, some participants stated their
experience was more engaging ("With the blocks, you engage more, you have a greater
126
engagement, because everything is ready, you do not have to assemble. It generates a greater
engagement" (P12:11)).
Additionally, a participant felt he had more time to explore ideas ("It is more ready, so give more
time to explore things." (P02:8)), which is supported by another participant ("It's ready, and I use
it faster" (P16:13)).
One participant felt that there was a good balance in adaptability and readiness (the perception
that the system presents well-defined structure, closer to a final product) of Probatio ("So this
balance is cool: it's very customizable and, at the same time, it's very ready too. You can quickly
fit it" (P16:10)).
One participant mentioned that Probatio was robust ("The block I felt it robust, I did not feel it
fragile. The weight, the way they are stuck I felt something rigid, something firm. I did not feel it
could break" (P07: 27)).
8.3.2.7. Restricted Space
In the opposite direction, because of the limited length of the wires in GSToolkit, depending on
the number of sensors, the interaction could become difficult ("The extension of the wire, then
you start to get a lot of things close" (P07:22)) in different aspects, such as restricted space,
fragility, disorganization, lack of fixed place.
Firstly, in terms of space ("So, I found it less ergonomic because everything is too tight" (P02:1)),
the component tends to be restricted by the length of the wire, and close to the breadboard.
The connection between the wire and the connectors is only an insertion with no additional lock
mechanism, and it was considered by one participant as fragile ("So, the wire is fragile. If you
move, with the touch it will release and disconnect there [in the GUI]." (P07:21)). With that, the
risk of making a mistake increases ("The way the sensors were, they're open, so any bullshit
you pull a wire, you have to put it back." (P10:3)).
8.3.2.8. Disorganized, Fixed in Place and Musical Interaction
Because the sensors were hanged by the wires and with no additional physical support, some
participants mentioned that the setup become disorganized ("The big problem is that things get
disorganized, hang by the wires." (P01:3)), directly impacting on the usability of GSToolkit ("The
fact that things are hanging by wires makes the usability it very difficult" (P01:5)). On the other
hand, one participant said the set up with Probatio was more organized ("It was much easier,
more organized for you to use." (P13:5)).
Without the physical support, the sensors tend not to be fixed in a certain place ("Each time you
use, those sensors will be in different places." (P10:8)), which may have caused difficulties in the
interaction ("To interact, things are loose on the table, so it was bad to maneuver, to interact with
the sensors." (P13:4)). One of the consequences may be the interference in the musical
127
precision ("But the thing of them being thrown to the side of the board, they end up becoming
less accurate musically speaking." (P14:23)).
Regarding Probatio, one participant mentioned that the structure could benefit musical interaction ("So when you have the controls already stuck rather than loose with the wires,
musically speaking, to get into musical notes, I found the blocks better" (P14:24)).
8.3.2.9. Reduced Dimensions
Due to the reduced dimensions of GSToolkit’s components, some participants mentioned that
it was difficult to assemble the circuits (P01, P02, P06, P08, P13, P14, P15, P16).
They stated that the wires and connectors were small ("That fucking little wires, you want to get
to the sound!" (P16:5)), the breadboard was small ("The little board being small makes it a bit
difficult" (P14:9)), and the size of the labels on the board was small ("You keep looking at these
little letters." (P02:3)).
8.3.2.10. Long Time
Because of the reduced dimensions, some participants felt they made some mistakes ("In the
case of the sensor, I took off my glasses so I could see right and put it in. I was wrong sometimes"
(P15:12)), and sometimes felt confused ("I found the two similar, but with the sensors I had more
difficulty because of the connections on the Arduino. It confused me a little." (P18:1)). As a result,
some participants felt they were losing time ("I put the pin in the wrong place, I lost time with it."
(P13:2)).
Additionally, reading the instruction card also interfered with the time spent to build the
prototype in GSToolkit ("Because it takes some time for you to read the card, it takes some time
for you to pick up those little things and connect ..." (P16:4) and "The Arduino's for me was slower
because I had to search the card to find out where are the connections." (P01:8) and "You spend
some time to see this thing: how do I connect it? Where is it? "(P02:2)).
All of these individual comments may lead to an overall perception of difference duration to
achieve results in the two systems ("We take a lot more time to test than with the blocks." (P13:3)
and "I would say that with the sensors you take longer." (P14:25)).
With GSToolkit, some participants mentioned that the time to achieve something satisfying was longer ("But the working time is much higher for us to generate a cool thing." (P08:21) and
"it took me a long time to produce more sound results" (P17:3)). Probably, one of the causes is
the effort of performing numerous tasks to add only one input parameter ("The tricky thing
about it [the sensor interface] is because there are many connections you have to make to use a
single sensor." (P11:4)).
This excessive time to achieve a musical result might have caused a feeling of frustration in
some participants ("Maybe at most with the sensors I have felt a little frustration about the work
of assembling." (P19:9)). And, as quoted before, also anxiety ("That fucking little wires, you want
128
to get to the sound!" (P16:5)). In some cases, even distress ("Because what interests me is the
sound, I want to see the sounds right away... And then those stages of having to connect the
wires and having to be right, it will take longer and cause a little more angst." (P12 :5)).
Because of that, some participants mentioned that the approach required to deal with GSToolkit
should be different, calmer ("Actually, [with] the Arduino you have to be calm. It's another timing."
(P08:5)).
However, despite all of the negative comments, some participants find that GSToolkit was
engaging and funny to use ("I had a lot of fun with this one [the sensors]. I would spend the
afternoon traveling around here [sensors]." (P02:18)). This might be explained by two different
kinds of engagements: technical vs. musical, that will be discussed later.
8.3.2.11. More Parameters in Just One Hand
In both systems, participants seem to want to control more parameters with just one hand
("Sometimes you want to interact with three things at the same time." (P13:8)).
The lack of a physical support in GSToolkit causes the participants to use one hand for support and the other for manipulation ("I had to use both hands for each one." (P01:12)). The participants
expected to use one hand to control more than one musical parameter, and with GSToolkit it
seemed to be impossible ("In the second [the sensors], it is almost impossible to touch and have
the same hand to generate various information" (P08:27)).
Although some participants found that Probatio was better to use than GSToolkit ("Physically,
they [the blocks] are more comfortable to use." (P03:13)), the system caused frustration to other
participants due to the difficult interaction of using one hand to control various parameters ("It
was more of a frustration in the matter of trying to coordinate the commands there [...] I wanted
to do several things at once, and I could not do it" (P18:9)).
One participant explained that this limitation is due to the distance between the blocks ("In the
blocks, this distance between one object and another ... sometimes makes it difficult, for example,
to create forms of performing [...] by taking advantage of the movement of a single hand"
(P08:36)). The same participant even attempted to stretch his hands with no success ("And I
already have a big hand, I already had to stretch a lot and sometimes I could not." (P08:26)). He
concludes that allowing the user to control more parameters with just one hand may contribute to
developing a performance repertoire of gestures for some combinations of blocks ("Because
then it would end up being so experimental and you would already start to create a culture of
performance." (P08:35)).
8.3.2.12. Easy and Quick
The quick way of connecting the blocks was positively highlighted by fourteen participants (P01,
P02, P04, P07, P08, P09, P10, P12, P13, P14, P15, P16, P17, P18). Some participants
commented that they could reach musical experimentation with less time ("Here [the blocks] I
129
move to solve musical things faster" (P02:19) or "I felt the sound results of my connections more
quickly" (P17:17).
There seems to be an initial barrier to using GSToolkit ("Actually, it's difficult to start using. It's
not of using, it's to start using." (P09:6)). One participant mentioned the assemblage as a barrier to achieve musical results ("With the sensors, you get this feeling, that you first have to make a
connection that has nothing to do with sound." (P.12:6)), and he concludes Probatio might reduce
this barrier ("It is a feeling of not having to make an assembling before you start to make sound"
(P12:7)). Other participant did not consider GSToolkit boring, but that it took longer to test musical interactions ("This thing of having to connect the wires, it's fun too, but it takes longer."
(P12:4)).
One participant also mentioned that quality of the musical results with the blocks would be
higher (“I would get better and faster results with the blocks" (P12:14)). Related to this, other
participant said that one could reach a higher level of musical experimentation with Probatio
("The blocks have already been more practical, artistically, they already comprise languages, try
languages, try clichés, try performative musical organizations" (P17:20)).
One participant made a correlation between the quickness and the creative flow, mentioning
that it was easier to reach a creative state ("For me, I felt ... working faster, which then lets
creativity flow more easily with the blocks" (P14:12)).
8.3.2.13. Urgency in Reaching Musical Results
One participant highlighted the urgency of the musician in general to reach a musical result ("At
the time that we are creating, at the time that we are composing something, we want to touch C
and play C" (P18:6)). It is a creative necessity ("So the faster you can do what you imagine, the
better." (P14:7)).
Another participant said that was easier to have a musical experience with Probatio than with
GSToolkit ("For me, the musical experience was better on the blocks because I got to get to a
musical thing more easily." (P14:22)).
Some participant mentioned that they could reach musical experimentation faster with Probatio
("You jump right into the sound issue, musical." (P02:22) and "I think that with the blocks I could
go more directly to what I wanted to do." (P18:2) and "On the block, it was a most immediate thing
to fit in and get started, to get somewhere "(P17:5)).
Participants mentioned that Probatio was not only fast to build prototypes ("I have tried many
more things and gained much more" (P12:16)) but also to modify them ("[In blocks], it is much
easier to modify." (P10:15) and "The other was one more like putting, and it did not work, I would
change the other, and I would try it until I found the cool one" (P07:7)). Other participant compared
his experience with both systems and highlighted that placing and removing a block was easier than managing the cable connections ("With the blocks, due to the ease of experimentation.
You put a block, it did not work, you remove it, you put another," ah, here I can control what I
130
want to control", so it's easier than you take four wires and put four more. "(P11:12)). Another
participant summarizes ("Everything is easier to put on, to test." (P13:17)).
Because of that, some participants seemed to enjoy the interaction ("With the blocks, this thing
is much faster. Thought, imagined, picked up, fit, it already appears on screen, that's really cool."
(P14:10) and "I liked the quick connectivity." (P08:1)). Besides, some considered that the musical experimentation was better with Probatio ("There is the musical engagement, of the musical
interaction, which was better with the blocks because I arrived faster" (P16:16)). With Probatio,
some participants reached a sound result more quickly ("I felt more quickly the sound results
of my connections" (P17: 17)). A participant summarized his impression of Probatio if he could
use it for a musical presentation ("Imagine and quickly assemble." (P14:21)).
One participant was excited to describe the easy and quick use of Probatio ("I found it all very
intuitive. The blocks I found to be fantastic, phenomenal. The duration [of the session] makes you
want to spend more time experimenting" (P07:4)). Other participant mentioned that Probatio is
efficient, effective, feasible ("In the blocks, it is much more efficient, much more effective, much
more possible." (P12:17)).
Due to a more direct approach towards musical interaction, one participant concludes that the
blocks would be interesting for musicians ("At first, for a musician, the block is much more
interesting." (P13:10)).
8.3.2.14. Cognitive Load and Creative Flow, Kinds of Engagement
Five participants (P02, P09, P13, P16, P17) explained they felt two kinds of engagements while
using the systems ("So for me, I feel engaged by the two. Now, for different environments,
different circumstances." (P17:13) and "I have been involved with both of them, but they are
different engagements" (P02:17) and "Then the two gave me different engagements" (P07:16)).
Other participant contrasted the engagement of assembling something in general and the
musical engagement ("Because there is the engagement that is of the assembly, that is you
prepare the setup that you are going to use, this is very stimulating, very engaging. That was the
same in both. There is the musical engagement, the musical interaction, which was better with
the blocks because I have come more quickly." (P16:15)). One participant distinguished the two
experiences in two perspectives: aesthetic and structural, or artistic and engineering ("So this is
not an artistic, aesthetic issue, this is a more structural, engineering issue, so to speak." (P17:21)).
The sensors seemed to be more associated with the engagement of dealing with an assembly kit ("It has the kit thing for you to assemble" (P14:12)), and one participant highlighted the rational aspects of using it ("Making the sensor work, for me, is more rational than sensory.... It is, in my
view, more logical, rational" (P17:9)).
These engagements seem to differ in time ("It's another kind of involvement, which I also enjoy,
but it's a different time, a different timing." (P17:7)), and in objectives ("But the idea here [in the
sensors] is an idea of interface, the idea here [in the blocks] is an idea of sound." (P02:35)).
131
On the other hand, Probatio may have contributed for the participants to have a more intuitive experimentation ("I had a moment in the use of the X [the blocks] that I entered into another
mode. Let's say, a less cognitive and more intuitive way." (P02:26)). Here, intuitive is not related
to the ease of use, but to exploring sensorial stimuli instead of solving problems. He also
mentioned that when he reaches this intuitive level, he does not want to deal with technicalities ("And then I do not want to think about the connection anymore ... I want to do
something else, I want to put it there, map and feel it" (P02:30)). Another participant supports that
("Making the sensor work for me is more rational than sensory" (P17:9)).
Solving problems may be associated with GSToolkit ("But here I have to solve more technical
and ergonomic things" (P02:36)).
Probatio might have interfered in the perception of engagement of some participants ("I think if
one watches the video, I'm talking much less. I'm more absorbed in that process." (P02:32)), and
in the time perception ("The second moment [the blocks] passed much faster. I was really more
involved." (P05:3)). This seems to be related to the definition of “distortion of temporal experience”
described as one of the characteristics of being in creative flow (NAKAMURA;
CSIKSZENTMIHALYI, 2014).
In this dichotomy between engagements, one participant mentioned the Maker movement, and
associate it with the technical engagement in contrast to the musical engagement ("I had a good
deal of engagement in the two of them. In the first [sensors], more by a maker thing than a musical
thing." (P13:12)).
One participant said he was always aware if he had made some wrong connection or whether
everything is working properly ("You do not have to wonder if you have connected the ground in
the right place or the right slot for it to be recognized." (P01:14)). Thus, other participant mentioned
he was always alert ("In this guy here [the sensors], you are thinking all the time" (P02:33)). In
this context, the cognitive overload can probably become higher ("This option is a very
interesting option [the sensors], but it has this cognitive cost" (P02:4)), which is related to effort ("It has more intellectual wear [...] mentally demands more, I think" (P07:13)). Physical and mental effort ("It's fun, [...] but it's frustrating because you feel you have a more physical, strategic
clash there to have a sound." (P07:23)).
The constant change between these two kinds of thinking can probably interfere with the
creative flow ("So when you move to this other level something goes there and slaps you, and
you have to go back... Then your creative part gets stuck." (P02:31)).
The participant explicitly illustrated his experience with GSToolkit with the metaphor of jumping trees ("You always keep jumping from side to side" (P02:34)), where you mount the circuit and
tries to play it.
With Probatio, the technical encapsulation seemed to contribute to reaching musical
experimentation more quickly ("Here [the blocks] the technique you do not think very much. You
forget it fast... You jump at once for a sonorous, even musical question" (P02:16)).
132
One participant said it is important to not have interruptions during the flow ("That there really
is the moment that the system needs to be ready for the person to have a flow ... Because,
otherwise, you stop. And then, fuck, you come back to ... " (P02:27)). Therefore, it seems to be
important to divide the technical and the musical processes apart ("There's a moment that I
want to forget about this, and I want to know that I'm going to play the timbre, here I'm going to
close the filter, here I'm going to play the pitch ..." (P02:29) and “I do not want to think about the
connection ... I want to do something else; I want to put it there, map and feel" (P02:30)).
However, the technical engagement seems to be empowering ("But it's more empowering, for
me, you put the wire there and ‘fuck, it went wrong, it worked’, sense of belonging." (P15:17)),
and contributes to the sense of ownership or authorship towards the process ("You gain greater
possessiveness, a greater authorial feeling for what you are doing, because you are in a way
setting up the stuff." (P15:15)).
Besides, the sense of challenge may have engaged some participants in the technical part ("But I felt more engaged even with the sensors because I wanted to make it work, so I fought
harder for that to work than for the blocks." (P18:7)).
Furthermore, individual interests may also have played a role in the technical engagement
("[The sensors] get with other things [interests] of mine." (P07:14) and "I felt that I was very
engaged with this part [of the wires, the connections]. It is something I like. I stood there [wiggling
the arms simulating the assembly action of the wires and sensors] I wanted to explore a lot more.
"(P09:10)).
8.3.2.15. Bug, Errors, Feedback
Some participants felt frustrated with the loose contact issue in Probatio. As a consequence of
this error, they lost all their mapping strategies after the error ("I got frustrated with some crashes,
the things that came loose and lost contact ..." (P16:20)). The mapping software, Webmapper,
did not provide a way of saving the connections or recovering from a failure, which forced the
participants to reconnect everything relying on her/his memory ("It falls, and you have to
remember." (P02:24)). Due to errors in Probatio, one participant stated that he felt angry when
he lost all the mappings (“Frustrated. Pissed off!" (P19:10)).
Regarding GSToolkit, the participants did not negatively highlight any major error during the
interview. However, one participant felt the lack of feedback indicating that the connection was
properly working or not ("Yes, sometimes I made the connections, and it did not work!" (P18:8)).
A participant felt compelled to compare GSToolkit and Probatio saying that he did not face any system error with the former ("And the sensors no. The drawing is there, I fit it, and it did not
cause a bug." (P03:5)).
133
8.4. Discussion With both quantitative and qualitative evaluation, we could assess user’s interaction following two
aspects respectively: (1) external observation, and (2) user’s perception about the systems. In
this section, we attempt to correlate both results. Comparing Probatio and GSToolkit helped to
understand the user’s engagement during the interaction with a prototyping toolkit for DMIs. The
systems differ in two major aspects: (1) physical structure and (2) way of connecting the modules.
Considering our first three hypotheses (H1, H2, H3 in Figure 8.2), the results confirmed that: (1)
the duration to achieve a functional prototype was shorter in Probatio and longer in GSToolkit; (2)
the cycles of idea exploration was more numerous in Probatio and less numerous in GSToolkit;
(3) the number of distinct items used was greater in Probatio and lesser in GSToolkit. However,
regarding the overall user experience, the results demonstrated that there were different kinds of
engagements with the systems, which did not confirm our expectation that the participants would
not engage with GSToolkit, feeling more frustrated with this system. Additionally, we did not
expect the various errors that happened in Probatio, and we expected that the participants would
have many more errors in GSToolkit, what did not confirm.
8.4.1. About Probatio
Both quantitative and qualitative results suggest that Probatio provided a quicker and easier way
to reach musical interaction (as shown in section 8.3.1.1.2 and 8.3.2.12). Besides, if we consider
that the mounting phase is a technical barrier that has to be overcome to enter the cycles of
mapping and testing, which are the actual exploration of musical interactions, Probatio’s shorter
mounting phases and longer testing and mapping phases (see section 8.3.1.1.2) may indicate
that the system reduced the implementation barriers and fostered the musical experimentation.
Qualitative results suggest that Probatio’s physical support presented a good balance in flexibility
and immediate usability (section 8.3.2.6). With it, the participants reported had more time to
explore ideas, and their experience was commonly mentioned as engaging. Additionally, the
number of components and the easy way of connecting them caused the participants to feel that
they experimented more combinations in less time. This is supported by the higher number of
distinct items (section 8.3.1.1.4) used in Probatio in comparison with GSToolkit which suggests
that the system allows a broader exploration of the possible combinations.
In the context of designing construction kits for kids, Resnick and Silverman (2005) propose the
following guideline: “low floor, high ceiling, wide walls”, respectively meaning that the kit should
be easy for beginners, allow an increase of complexity for more experienced users, and permit
the exploration of different directions based on creativity and imagination (RESNICK;
SILVERMAN, 2005). The results suggest that Probatio might fulfill the low floor, and wide walls
guidelines. We believe that, for assessing high ceiling, we would need to define longer periods of
experiment both duration of sessions and multiple sessions over weeks or months.
134
Participants using Probatio mentioned that the number of items could be overwhelming because
it was too much information to process at once (section 8.3.2.1). It seems to be contradictory to
the fact that, actually, the number of individual components of GSToolkit was higher. A possible
explanation is that, visually, the volume of GSToolkit’s sensors is smaller than the blocks and
supports of Probatio. Besides, the presence of instruction cards probably attenuated the amount
of information to process, since the stack of cards presented a sequential way of exploring the
space of possibilities.
Additionally, they also indicated some shape limitations in Probatio, mentioning that forms were
not ergonomic due to the pointy edges, and cubic shapes, and also that some of the components
did not fit together (8.3.2.5). Besides, the distance between blocks caused the participants to
stretch their hands to perform some gestures, as they wanted to control more than one input
parameter with just one hand (8.3.2.11). They said that possibilities in controlling more
parameters with one hand could probably influence the formation of a gesture repertoire for blocks
in Probatio.
Additionally, the presence of a physical structure contributed to the perception of robustness of
Probatio (8.3.2.6). Besides, because the components rested in fixed places, the participants could
perform more accurate gestures, which benefited the musical interaction (8.3.2.8).
These points highlight the importance of experimenting with the physical structure in a flexible
way. Normally, the majority of the prototyping tools (see section 4.2.1) focuses on providing
sensors and actuators, but the designer has to deal with building the suitable physical support.
We find it is a hindrance for idea exploration in DMI design since the designer keeps changing
the contexts of attention. According to Sadler (2016b), technical interruption negatively affects
the creative thinking (SADLER, 2016).
Some participants highlighted the resemblance of the physical support with existing musical
instruments, and maybe because of that, it yields attraction for the object (8.3.2.6). The video
analysis shows that with Probatio participants experimented different ways of holding the
prototype (8.3.1.1.5). It is difficult to state that this happens due to the signifiers present in the
objects. However, it seems to be a promising feature for further development in the future.
8.4.2. About GSToolkit
Participants indicated that GSToolkit process induced them to follow a certain logical path, which
is mainly related to reading the instructions cards (8.3.2.2). The cards seem to have attenuated
the learning curve, saving user’s energy on understanding how the system works.
Different from Probatio, participants mentioned that GSToolkit had a great potential for
customization, allowing the users to arrange the position of the items with more freedom in the
desired way (8.3.2.4). Additionally, the wire manipulation appeared to have caused a good
sensation of authorship and made the participants in control of the process.
135
In fact, the authorship and ownership feeling were related to both systems, and this appears to
be in accordance with other result reported in the literature: the co-design experience in
Angesleva et al.’s (2016a) study. The authors conclude that the user seems to give more value
to a self-made product and the experience of creating or modifying a product “generates a sense
of creativity and enjoyment in the user, in the accomplishment of a task." (ÄNGESLEVÄ et al.,
2016b).
About the negative aspects of GSToolkit, it was often mentioned that the laborious way of
connecting the items was a barrier to achieving musical results (8.3.1.2.3 and 8.3.2.2). This
caused the participants to think before building a prototype. In the context of designing physical
and virtual interactions, Wiethott et al. (2012) argue that the design process should focus on the
concept of “working it through rather than thinking it through”, (WIETHOFF, 2012). In our
experiment, the “thinking it through” turned the participants to be more focused, and ended up
causing them to feel that they experimented fewer options - a fact that is confirmed by the
quantitative results.
Furthermore, the lack of physical support caused the sensors to be hanged by the wires, and the
interaction was limited to an area closed to the breadboard (8.3.2.8). It made the environment
disorganized, what we can assume that affected the usability. The components did not rest in
fixed positions, and one of the user’s hands had to be always busy as a support for the
components. The video analysis shows that most of the participants were curved towards the
sensors, and the only position tested was the tabletop (8.3.1.1.5).
Due to the fragile connections and the reduced dimensions of the header connectors (8.3.2.7 and
8.3.2.9), the risk of making mistakes made the participants be always aware of their actions, in
constant alertness. This caused the sensation of losing time with actions that did not relate to
musical experimentation. The numerous tasks for achieving just one input result made the
process longer. This caused anxiety, frustration, and even distress in some participants.
8.4.3. Summary of Bugs and Errors
After the changes we made after the technical pilot test, we did not expect the high number of
bugs that happened in Probatio during the experiment sessions. However, mainly due to the
overall positive impressions, we believe that the error did not play a major role in the user’s
interaction with the system. In its turn, we expected that the users would have much more
problems with GSToolkit, because of the possible wiring mistakes. However, in general, the users
could manage to perform the wire connections, solving the problems themselves without the need
of external help. This might be due to the presence of instructions cards, which provided a quick
visual reference for the user.
136
8.4.4. Different Engagements
The participant’s perception of engagement was twofold: the technical engagement of assembling
parts of a system, which is more related to rational aspects of solving problems, and the musical
engagement, which is associated with the intuitive exploration of artistic aspects of sound and
musical interaction. These engagements differ in time, objectives, and nature. This dichotomy
seems to be related to the two thinking styles defined by Epstein (1996): one intuitive-experiential,
and the other analytical-rational (EPSTEIN et al., 1996).
The shift between the two was reported to affect the awareness of the creative flow. Participants
mentioned that they engaged technically with both systems, but highlighted that felt more
musically engaged with Probatio. This probably happened due to the technical encapsulation and
the ability to rapidly obtain musical results.
Another point is that the more numerous transitions of mapping and testing in Probatio imply more
cycles of idea exploration, which according to Camburn (2015a), Beaudouin-lafon (2000c), and
Von Hippel (2001) can lead to design maturity and trial-and-error learning (CAMBURN et al.,
2015) (BEAUDOUIN-LAFON; MACKAY, 2000) (VON HIPPEL, 2001). In fact, participants
commented that they could accomplish higher levels of musical experimentation with Probatio,
resulting in better musical results. Indirectly, we may conclude that with Probatio the users had a
better understanding of the how the items could be combined and used.
8.4.5. Three Profiles
A possible interpretation of the results led us to three categories of user’s profiles. Although these
profiles might not be generalizable to other contexts, we will use them as a conceptual scaffold
to deepen our discussion.
The first profile has the customization of the instrument as a priority, focusing on the freedom to
use the sensors, to place them where they prefer and define the instrument structure by their
own. We will call this group the builders.
The second group is characterized by concentrating on musical experimentation and the
combination of input devices with sound outputs, trying to identify elements that fit together. This
group will be called experimentalists.
The third group has more focus on virtuosity, i.e. on the instrument's ability to provide precise,
nuance-controlled gestures that allow more than one parameter to be controlled with just one
hand. The name of this group can be virtuosi.
We interpret that these three groups are not exclusive, that is, a single participant can be related
to these three categories.
For the builders, what seems to be the most important is the concept of a white box, or transparent
box, as opposed to the black box, or the technical encapsulation (discussed by (SADLER et al.,
2016a)). The main motivation is to reach an instrument made by the hands and to be proud to
137
understand everything that was done in the development of that instrument. This case seems to
relate to the feeling of being a mad scientist, a Gyro Gearloose, dealing with wires, going in the
intricacies, the details of the instrument. For this first category, Probatio seems to draw less
attention than GSToolkit, even though the latter demands more development time than the
former. The structural and functional constraints of Probatio seem to evoke a sense of already-
defined instrument. While GSToolkit, even in potential, provides a sense of unfolding the world
not yet explored. GSToolkit appears to be more challenging, and this may be interesting for
people with more skills, but discourages those with fewer skills. Here comes an interesting point
about layers of abstraction, since GSToolkit itself is already an encapsulated version of the raw
sensors. What is the ideal balance between technical encapsulation and freedom of
implementation?
For the experimentalists, what matters is to achieve the musical result in the most immediate way
possible. There seems to be an anxiety in musical experimentation. The quick connection of the
Probatio suited this group better. With Probatio, participants were able to easily and quickly
explore combinations of gesture control and sound output, while GSToolkit imposed some
barriers.
For the virtuosi, the main thought seems to be how to achieve maximum fine-grained parameter
control with just one hand. Also, we could see that some participants were interested in creating
a repertoire of gestures, somehow start to think in the gestural point of view and not only in the
prototype development. This fact seems to relate to a discussion raised by Vertegaal et al. (1996),
in which the authors defend that the digital instruments should allow that their properties can be
“frozen” to provide ways to develop gestural techniques for its use (VERTEGAAL; UNGVARY;
KIESLINGER, 1996). For this third category, GSToolkit did not seem to be interesting due to: (1)
the constant need to use a hand as a support for the sensors (as explained by (GUIARD, 1987)),
(2) because of the reduced interaction area, (3) because the sensors were not fixed and were
always disorganized on the table (thus, it makes difficult to practice muscle memory (LEVITIN;
MCADAMS; ADAMS, 2002)). On the other hand, the Probatio is not yet at a fine-grain level to
satisfy the cravings of this group. The blocks are still large, the hands are still far apart, and the
controls are still very unitary, i.e. an item can control only one parameter. We considered that to
please this group, the blocks need to be more ergonomic, smaller to be closer to each other and
with a greater amount of parameter controls on just one device.
8.4.6. Limitations
A potential criticism of our experiment can be related to the following topics, which we present
with our comments below.
Comparison between two different systems: the difference between the systems is important
to stimulate the participants to compare unique features that are important for the development
of Probatio.
138
Duration of the sessions: we considered that the duration of each session was balanced
between schedule availability of the participants and the amount of data and information we
gathered to analysis. Perhaps, with longer duration, the number of participants would be reduced.
Number of sessions: it would be valuable to analyze the evolution of user’s experience over
time. For keeping the sample size, it would be necessary to make several toolkits available, which
would be currently unfeasible. Altering the sample size would reduce the depth of our results.
Sample size: for that, we followed the HCI literature recommendation and achieved a number of
participants that is considered reasonable.
Users’ gender: although trying to contact more female participants, women that matched our
profiling requirements are not numerous in Recife.
In this experiment, we focused on variables and impressions related to building a functional
prototype of a DMI. However, another direction would be on the resulting instrument. For future
experiments, it would be interesting to evaluate the outcomes taking into account the opinion of
external evaluators about the generated ideas of DMIs.
Although we focused on the individual interaction, from the pilot test, we could perceive the
potential of Probatio to be used in collaborative experimentation. It allowed that multiple users
could experiment at once. On the contrary, maybe the reduced area for assembling the circuit,
and perhaps the dimensions of the components, caused a reduction in the number of people
experimenting at once with GSToolkit.
8.4.7. Final Considerations
Considering our initial hypotheses, we realize that the use of Probatio reduced the assembly time,
increased the cycles of ideas exploration and also the time of this exploration. These specific
hypotheses of the experiment are directly related to our research questions, since by reducing
assembly time we are reducing the time and effort to build a prototype. Consequently, with shorter
duration, the cycles of exploration of ideas are strengthened.
As for the user experience, the experiment allowed us to access a piece of knowledge about the
various behavioral profiles regarding the construction of a DMI. Each behavior demands specific
system requirements.
We have seen that for builders, flexibility, freedom, and the feeling of being inside the process
are more important than immediate usability. For the experimentalists, urgency and immediate
musical outcome are more important than freedom and flexibility. For the virtuoso, the direct result
is important, but the interface needs more refinement.
The Probatio seems to be suitable for the “experimentalists”, but it proved inadequate for the
“builders” and still needs improvements to satisfy the “virtuosi”.
We can conclude that Probatio is not a general-purpose prototyping tool. The system has
limitations that at the same time serve as the initial path of exploring ideas. Its use is directed
139
towards a specific user behavior and at a particular stage of the design process. The toolkit seems
to relate better to a phase where there is still no pre-established idea or to test simple ideas.
Surprisingly, the GSToolkit system that was developed to serve as a counterpoint in the Probatio
evaluation appeared to be interesting for part of the participants. The adjustment in the
encapsulation of technical details seemed to bring positive impressions.
The equivalence between the two systems and their use together during the prototyping process
can bring compelling possibilities to the various profiles of users' behavior. The user can start by
exploring early ideas with Probatio and, after the idea is consolidated, GSToolkit can be used to
expand this idea to levels that go beyond the structural limitations of Probatio. All of this is related
to the equalization of the levels of abstraction to maximize the exploration and concretization of
ideas. A restricted but faster start may encourage the exploration of ideas. With the idea defined,
the more focused use of a slower but more flexible tool is a good mix between agility and
suitability.
140
9. CONCLUSION As interest in designing interactive physical devices increases, it is important to refine the
methods and tools that improve the design process. In this perspective, artistic devices are useful
because their creation and development involve a variety of challenges.
This work focused on the cycles of idea exploration and prototyping as an approach to deal with
the complexity of DMI design.
9.1. Research Question Revisited 1) How to provide a structured and exploratory path for generating new DMI ideas?
To address this question, we followed one possible solution that was the adaptation of an existing
idea generation method, combining it with the concept of instrumental inheritance. The concept
leverages the familiarity and cultural hooks of existing instruments, and the method provides a
systematic and exploratory way to combine elements. We expect that this combination works an
ignition for generating new instruments ideas.
2) How to reduce the time and effort needed to build functional DMI prototypes?
Considering the second question, we propose a modular toolkit that embedded the
aforementioned concept and design method, aiming to provide the DMI designer with ways of
achieving physical, functional prototypes faster and with less effort.
Probatio’s evaluations quantitatively confirmed our objective of reducing time and effort to achieve
functional prototypes. Also, qualitative results indicate the suitability of Probatio in the initial
phases of design, for users interested in rapidly generating and evaluating musical interactions
ideas. The users appear to appreciate the value of Probatio as a tool for the fast design of
functional prototypes, allowing them to realize their musical interaction ideas with a low entry
barrier.
The results led us to believe that we have advanced in the exploration of our research questions
by understanding possible ways to provide structure and exploratory steps for idea generation
and to reduce the time and effort of building functional DMI prototypes.
9.2. Contributions We consider that our main contributions to the DMI design are:
• The introduction and exploration of the concept of instrumental inheritance: we
presented the concept of transferring structural or gestural elements from existing
instruments as an initial constraint to ignite the creative process for generating new
instruments ideas. Based on common knowledge and existing cultural hooks, the new
instruments can leverage the existing intimate relationship musician-instrument and provide
141
the audience with familiar elements, avoiding disconnection and perhaps more engagement
during the performance.
• The development of a prototyping toolkit that embeds an idea generation method and a concept: Since the toolkit was built based on the method and the concept, we believe it
has the potential to transmit the inherent knowledge through the artifact and allows the
designer to easily and quickly generate, implement, test, and modify her ideas. Supported by
evidence in the literature, with the increasing number of cycles, we expect that designers and
users can achieve adequate results considering user’s needs, intentions, and contexts of use.
We believe that our originality resides on the literature’s lack of structured and integrated
methods and tools for DMI design that helps the designer to generated ideas and immediately
evaluate them with a functional prototype.
As additional contributions, we can mention:
• The use of morphological chart in DMI design: to our knowledge, the morphological chart
based on instrumental inheritance is a novel approach towards the design of DMIs. We
believe that the other idea generation methods can be further explored to foster new
exploratory paths in DMI design.
Focuses on boosting the cycles of idea exploration and prototyping during the design process
and comprises, ultimately, we hope that our approach can contribute on the cycles of musical
instruments’ evolution.
With further development, we also expect that the concept, method, and toolkit can contribute not
only to DMI design but also to the conception and quick testing of interactive physical objects in
general, since the approach of analyzing existing products and representing their parts as
physical real-time-reactive easily combinable building blocks may provide alternatives to boost
design cycles.
9.3. Limitations Although our work is mainly focusing on exploring the gestural controllers and mappings for DMI
design, there is still elements that can be particularly developed in these regards.
Concerning mapping strategies, the graphical user interface based on Webmapper do not allow
the combination of input values, for example, trigger a value if the other passes a threshold. This
fact seems to reduce the experimentation to simple interactions.
Moreover, we did not go further in considering other important aspects that constitute the DMI,
such as feedback and sound module. We consider that Probatio still misses important visual and
haptic features that could contribute to a better user’s experience. Besides, the options of sound
production.
Considering the evaluation, both cycles assessed users that did not have previous knowledge of
the systems and spent a small amount of time using them. This might have caused the users to
142
maintain a superficial level of interaction, focusing on simpler elements of the systems. With that,
we may have a limited understanding of some aspects of Probatio.
Specifically, Probatio showed its potential suitability for the profile of experimentalists, which focus
on immediate sound results. However, the toolkit demonstrated its restrictions for the builder’s
profile, given the demand for more freedom to define the structure and position of the sensors. In
addition, for the virtuosi’s profile, Probatio lacked elements that could provide the development of
a gestural repertoire and execution techniques.
9.4. Future Works From the point of view of the concept, we believe that instrumental inheritance, both from a
gestural and structural point of view, can be used in other areas as a lens for generating
alternatives. An area whose application seems to be suitable is in the design of complex
interaction artifacts ((RAMALHO, 2017)). This class of artifacts is based on the high degree of
dexterity and gestural evolution required to perform tasks. Some examples are tools for surgery,
aircraft cockpits, game controls, Formula 1 steering wheel. Therefore, gestural inheritance
appears to be a promising approach to leverage existing techniques, transferring them to new
artifacts.
Considering the method, the morphological chart we built can be expanded by further exploring
functions to find more details that can be explored to generate ideas for new instruments. Due to
our iterative and incremental methodology, a possible postmortem interpretation is that the chart
ended up being built as an intermediate stage because our main focus was on the development
of the prototyping toolkit. In fact, for our purpose of providing an immediate way for the DMI
designer to achieve functional prototypes and rapidly test them, the morphological chart seems
to be a means and not an end.
There are many fronts that can be explored in the technical evolution of Probatio. One of these is
ergonomic suitability to provide a better user experience. Instead of using cubic shapes with
pointy edges, it may be interesting to try out more rounded or organic shapes that can fit the
user's hands. In addition, to allow better manipulation, the dimensions of the blocks can be
reduced. This demands a work of miniaturization of the electronic components, which we do not
see as impossible or difficult, but it just was not the focus of this work.
Although the Probatio was well evaluated for its robustness, the MDF proved to be an unreliable
material due to variations in dimension with air humidity. In addition, over time the material
becomes porous. Therefore, it is interesting to evaluate alternatives such as applying a coating,
paint or use alternative materials such as metal or plastic.
Furthermore, we can think of a graphical interface more integrated with the physical forms of
Probatio. An interface that in real time identifies and represents the shape of the physical world
in the virtual world. We believe this would facilitate the next steps in the design process. Let us
imagine that after defining a satisfactory configuration of blocks and supports, the user could
143
generate the plans of a laser cutting with a simple push of a button. The system would also provide
a list of materials and a tutorial explaining the step-by-step construction of that one non-modular,
"frozen" version of the prototype. In this way, the user could begin to create a gesture repertoire
for this instrument-specific instantiation. Perhaps this approach fits the profile of virtuosi discussed
earlier.
In addition to Probatio's technical development details, we can reflect on the appeal of
incorporating a certain type of knowledge into the object. To access this knowledge, it would be
necessary to have a high level of technical expertise. The use of sensors, actuators, and
microcontrollers in the music still has a lot of space to be explored, but the entry barrier may deter
users whose focus is not technical but rather artistic. On this side, we have the musicians with
their individual needs, intentions, and contexts of use specific to their artistic expression.
Probatio's role is to unite access to the technical world without losing the focus on the creative
path. It is the encapsulation of the technical details to allow a quick and direct access to the
functionalities. We believe that this exploration of new knowledge can reconfigure the musician's
initial ideas since we can consider it as a cognitive expansion.
Let us look at a possible use of Probatio as a knowledge transmission tool. Let us say there is
Alice, who is a designer of DMIs, and Bob, who is a musician who is interested in new interfaces
for musical expression. After scheduling a meeting, Bob visits Alice's workshop. Bob has his own
artistic intentions and already comes up with some ideas of instruments that he would like to build.
Alice disposes the Probatio on the table, and Bob begins to explore the possibilities of
combination. At this point, Bob begins to discover types of sensors and input devices he had not
yet known. With the freedom to map any input into the outputs, Bob begins to reconfigure his
initial ideas of possible instruments. Alice has a role as simply mediator or facilitator (in fact, at
this early stage, Alice may even be removed from the scenario) because it is as if the object
already communicates the possible paths of exploration and passes quickly a knowledge that
would take a long time for Bob to access alone. Despite being interested in the technical parts,
Bob really feels fulfilled with the musical interaction, which is his main focus. Finally, when feeling
satisfied, Bob then defines a set of combinations that most pleased him. These combinations can
be related to structure, gestural controllers or mapping strategies. With this set, Alice now can
explore with Bob the more specific possibilities for his context and intentions. In this example, the
benefit of using Probatio is to clarify ideas and reduce the time to come up with an interesting
prototype for the musician. Perhaps, without using Probatio, Bob would not have been able to
form ideas that were not in his head and immediately test them and check their suitability for his
context and intention. The Probatio as an experimentation catalyst.
144
REFERENCES ACEITUNO, J. Direct and expressive interaction on the desktop: increasing granularity, extent, and dimensionality. [s.l.] Université Lille 1, 2015.
ADIKARI, S.; MCDONALD, C.; CAMPBELL, J. Little Design Up-Front: A Design Science
Approach to Integrating Usability into Agile Requirements Engineering. In: JACKO, J. A. (Ed.). .
Human-Computer Interaction. New Trends: 13th International Conference, HCI International 2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, Part I. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2009. p. 549–558.
ALLISON, J. T.; PLACE, T. Teabox: A Sensor Data Interface System. Proceedings of the 2005
International Conference on New Interfaces for Musical Expression (NIME05).
Anais...Vancouver, BC, Canada: 2005
ÄNGESLEVÄ, J. et al. The Results of Rethinking Prototyping. In: Rethink! Prototyping. Cham:
Springer International Publishing, 2016a. p. 201–210.
ÄNGESLEVÄ, J. et al. Beyond Prototyping. In: Rethink! Prototyping. Cham: Springer
International Publishing, 2016b. p. 161–199.
BALLAGAS, R. et al. iStuff: a physical user interface toolkit for ubiquitous computing
environments. Proceedings of the SIGCHI conference on Human factors in computing systems, v. 16, n. 5, p. 537–544, 2003.
BANGOR, A.; KORTUM, P. T.; MILLER, J. T. An Empirical Evaluation of the System Usability
Scale. International Journal of Human-Computer Interaction, v. 24, n. March 2015, p. 574–
594, 2008.
BANZI, M. Getting Started with Arduino. [s.l.] “ O’Reilly Media, Inc.”, 2009.
BARBOSA, J. et al. Towards an evaluation methodology for digital music instruments considering performer’s view: a case study. Proceedings of 13th Brazilian Symposium on
Computer Music. Anais...2011
BARBOSA, J. et al. Considering Audience’s View Towards an Evaluation Methodology for Digital Musical Instruments. Proceedings of the International Conference on New Interfaces for
Musical Expression. Anais...2012
BARBOSA, J. et al. Designing DMIs for Popular Music in the Brazilian Northeast : Lessons Learned. Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...Bâton Rouge, US: 2015a
BARBOSA, J. et al. What does “ Evaluation ” mean for the NIME community? Proceedings
of the International Conference on New Interfaces for Musical Expression. Anais...Bâton Rouge,
US: 2015b
BARRACLOUGH, T. J.; MURPHY, J.; KAPUR, A. New Open-Source Interfaces for Group
145
Based Participatory Performance of Live Electronic Music. (B. Caramiaux et al.,
Eds.)Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...Goldsmiths, University of London, UK: Goldsmiths, University of London, 2014
BASALLA, G. The Evolution of Technology. [s.l.] Cambridge University Press, 1989.
BATTIER, M. Electronic Music and Gesture. Trends in Gestural Control of Music, n. 1920, p.
328–330, 2000.
BEAUDOUIN-LAFON, M. Instrumental interaction: An Interaction Model for Designing Post-
WIMP User Interfaces. Proceedings of the 18th international conference on Human factors in computing systems - CHI ’00, v. 2, n. 1, p. 446–453, 2000.
BEAUDOUIN-LAFON, M.; MACKAY, W. E. Prototyping Tools and Techniques. Prototype Development and Tools, p. 1–41, 2000.
BENTLEY, J. Programmimg pearls. Communications of the ACM, v. 28, n. 9, p. 896–901, 1985.
BERDAHL, E.; JU, W. Satellite CCRMA: A Musical Interaction and Sound Synthesis Platform. Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...Oslo, Norway: 2011
BERTHOLDO, A. P. O. et al. Agile Usability Patterns for UCD Early Stages. In: MARCUS, A.
(Ed.). . Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience: Third International Conference, DUXU 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part I. Cham:
Springer International Publishing, 2014. p. 33–44.
BEVILACQUA, F.; MÜLLER, R.; SCHNELL, N. MnM: a Max/MSP mapping toolbox.
Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...2005
BIRNBAUM, D. et al. Towards a dimension space for musical devices. Proceedings of the
2005 International Conference on New Interfaces for Musical Expression (NIME05).
Anais...Vancouver, BC, Canada: 2005
BLAINE, T.; FELS, S. Collaborative Musical Experience for Novices. Journal of New Music Research, v. 32, n. 4, p. 411–428, 2003.
BLAINE, T.; FORLINES, C. JAM-O-WORLD: Evolution of the Jam-O-Drum Multi-player Musical Controller into the Jam-O-Whirl Gaming Interface. Proceedings of the 2002
Conference on New Interfaces for Musical Expression (NIME-02). Anais...2002
BONGERS, B. Physical Interfaces in the Electronic Arts: Interaction Theory and Interfacing
Techniques for Real-time Performance. In: WANDERLEY, M. M.; BATTIER, M. (Eds.). . Trends in Gestural Control of Music. [s.l.] Ircam Centre Pompidou, 2000. p. 41–70.
BONGERS, B. Electronic Musical Instruments: Experiences of a New Luthier. Leonardo Music Journal, v. 17, p. 9–16, 2007.
146
BONGERS, B.; HARRIS, Y. A Structured Instrument Design Approach: The Video-Organ.
Proceedings of the International Conference on New Interfaces for Musical Expression, n.
May, p. 18–23, 2002.
BRAUN, V.; CLARKE, V. Using thematic analysis in psychology. Qualitative Research in Psychology, v. 3, n. May 2015, p. 77–101, 2006.
BROOKE, J. SUS: A quick and dirty usability scale. In: Usability Evaluation in Industry. [s.l:
s.n.]. p. 189–194.
BROOKE, J. SUS : A Retrospective. Journal of Usability Studies, v. 8, n. 2, p. 29–40, 2013.
BROWN, T. Design Thinking. Harvard Business Review, v. 86, n. 6, p. 84–92, 2008.
BROWN, T. Change by Design: How Design Thinking Can Transform Organizations and Inspire Innovation. [s.l.] Collins Business, 2009.
BUCHANAN, R. Wicked Problems in Design Thinking. Design Issues, v. 8, n. 2, p. 5–21, 1992.
BUXTON, B. Artists and the art of the luthier. ACM SIGGRAPH Computer Graphics.
Anais...1997
BUXTON, B. Sketching user experiences: getting the design right and the right design. [s.l.]
Morgan Kaufmann, 2007.
CADOZ, C.; WANDERLEY, M. M. Gesture - Music. Trends in gestural control of music, p. 71–
94, 2000.
CALEGARIO, F. et al. A Method and Toolkit for Digital Musical Instruments: Generating Ideas
and Prototypes. IEEE MultiMedia, v. 24, n. 1, p. 63–71, jan. 2017.
CAMBURN, B. et al. A Systematic Method for Design Prototyping. Journal of Mechanical Design, v. 137, n. 8, p. 81102, 2015.
CAPJON, J. Engaged collaborative ideation supported through material catalysation. Nordic Design Research Conference, 2005.
CARD, S. K.; MACKINLAY, J. D.; ROBERTSON, G. G. A morphological analysis of the design
space of input devices. ACM Transactions on Information Systems, v. 9, n. 2, p. 99–122, 1991.
CLASON, D. L.; DORMODY, T. J. Analyzing Data Measured by Individual Likert-Type Items.
Journal of Agricultural Education, v. 35, n. 4, p. 31–35, 1994.
CONT, A.; CODUYS, T.; HENRY, C. Real-time gesture mapping in pd environment using neural networks. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...2004
COOK, P. R. Principles for designing computer music controllers. Proceedings of the
International Conference on New Interfaces for Musical Expression. Anais...2001
COOK, P. R. Real Sound Synthesis for Interactive Applications. [s.l.] A K Peters/CRC Press,
2002.
147
COOK, P. R. Re-Designing Principles for Computer Music Controllers: A Case Study of SqueezeVox Maggie. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...Pittsburgh, US: 2009
COOK, P. R.; SCAVONE, G. P. Audio Anecdotes: A Cookbook of Audio Algorithms and
Techniques, chapter The Synthesis ToolKit (STK) in C++. AK Peters, Natick, MA, 2004.
CROSS, N. Developments in Design Methodology. [s.l: s.n.].
CROSS, N. Engineering Design Methods: Strategies for Product Design. Third Edit ed. [s.l.]
John Wiley & Sons, 2000.
CROSS, N. Designerly ways of knowing. [s.l.] Springer, 2006.
CURTIS, A. Rhetoric of Flat Design and Skeuomorphism in Apple’s iOS Graphical User Interface. [s.l.] University of Rhose Island, 2015.
DAHL, D. W.; MOREAU, P. The influence and value of analogoical thinking during new product
ideation. Journal of Marketing Research, v. 39, n. 1, p. 47–60, 2002.
DAHL, L. Wicked Problems and Design Considerations in Composing for Laptop Orchestra. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...University of Michigan, Ann Arbor: 2012
DAHL, L. Designing New Musical Interfaces as Research: What’s the Problem? Leonardo, v. 49,
n. 1, p. 76–77, fev. 2016.
DOBRIAN, C.; KOPPELMAN, D. The E in NIME: Musical Expression with New Computer Interfaces. Proceedings of the 2006 International Conference on New Interfaces for Musical
Expression (NIME06). Anais...Paris, France: 2006
DONALD, M. A mind so rare : the evolution of human consciousness. [s.l.] Norton, 2002.
DOUGHERTY, D. The Maker Movement. innovations, v. 7, n. 3, p. 11–14, 2012.
DOW, S. How prototyping practices affect design results. Interactions, v. 18, n. 3, p. 54, 1 maio
2011.
DRUMMOND, J. Understanding Interactive Systems. Organised Sound, v. 14, n. 2, p. 124, 29
ago. 2009.
DUBBERLY, H. How do you design: A compendium of models. [s.l: s.n.].
DUBBERLY, H. The Space of Design. ACM’s Interactions, v. 17, n. 5, p. 1–9, 2010.
ELSEN, C. et al. Representation in early stage design: an analysis of the influence of sketching
and prototyping in design projects. ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, p. 737–747, 2012.
EPSTEIN, S. et al. Individual differences in intuitive-experiential and analytical-rational thinking
styles. Journal of Personality and Social Psychology, v. 71, n. 2, p. 390–405, 1996.
148
ESSL, G.; O’MODHRAIN, S. An enactive approach to the design of new tangible musical
instruments. Organised Sound, v. 11, n. 3, p. 285–296, 2006.
EXNER, K. et al. A transdisciplinary perspective on prototyping. 2015 IEEE International Conference on Engineering, Technology and Innovation/ International Technology Management Conference, ICE/ITMC 2015, 2015.
FALLMAN, D. Design-Oriented Human-Computer Interaction. Proceedings of the International Conference on Human Factors in Computing Systems (CHI’03), n. 5, p. 225–232, 2003.
FANTINATTO, R. I dream of wiresCanada, UK, USA, GermanyMonoduo Films, , 2014.
FEIL, M.; TUNG, F. Modular Toolkit Of Interface Components For Students To Efficiently And Easily Test Design Concepts. IDSA 2013 EDUCATION SYMPOSIUM. Anais...Chicago:
2013
FELS, S. Designing for intimacy: Creating new interfaces for musical expression. Proceedings of the IEEE, v. 92, n. 4, 2004.
FELS, S. S.; GADD, A.; MULDER, A. Mapping transparency through metaphor: towards more
expressive musical instruments. Organised Sound, v. 7, n. 2, p. 109–126, 17 jan. 2002.
FIEBRINK, R.; TRUEMAN, D.; COOK, P. R. A metainstrument for interactive, on-the-fly machine learning. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...Pittsburgh, US: 2009
FOUSE, A. S. et al. ChronoViz : A system for supporting navigation of time-coded data. CHI
2011. Anais...2011
FYANS, A. C. et al. Ecological considerations for participatory design of DMIs. Proceedings
of the International Conference on New Interfaces for Musical Expression. Anais...University of
Michigan, Ann Arbor: 2012
GAUNTLETT, D. The LEGO System as a tool for thinking, creativity, and changing the world. In:
LEGO Studies: Examining the Building Blocks of a Transmedial Phenomenon. [s.l: s.n.]. p.
1–16.
GELINECK, S.; SERAFIN, S. A practical approach towards an Exploratory Framework for
Physical Modeling. Computer Music Journal, p. 51–66, 2010a.
GELINECK, S.; SERAFIN, S. PHOXES - Modular Electronic Music Instruments based on Physical Modeling Sound Synthesis. 7th Sound and Music Computing Conference.
Anais...2010b
GERBER, E.; CARROLL, M. The psychological experience of prototyping. Design Studies, v.
33, n. 1, p. 64–84, jan. 2011.
GIBSON, J. J. The Ecological Approach to Visual Perception. New Jersey, USA: Lawrence
Erlbaum, 1979.
149
GILL, C.; SANDERS, E.; SHIM, S. Prototypes as inquiry, visualization and communication.
International Conference on Engineering and Product Design Education, n. September, p.
672–677, 2011.
GREEN, L. How Popular Musicians Learn: A Way Ahead for Music Education. [s.l.] Ashgate
Publishing, 2002.
GUIARD, Y. Asymmetric division of labor in human skilled bimanual action: the kinematic chain
as a model. Journal of Motor Behavior, v. 19, n. 4, p. 486–517, 1987.
GUREVICH, M. Designing for style in new musical interactions. Proceedings of the
International Conference on New Interfaces for Musical Expression. Anais...Pittsburgh, US: 2009
GUREVICH, M.; CAVAN FYANS, A. Digital Musical Interactions: Performer–system relationships
and their perception by spectators. Organised Sound, v. 16, n. 2, p. 166–175, 28 jun. 2011.
GUREVICH, M.; MARQUEZ-BORBON, A.; STAPLETON, P. Playing with Constraints: Stylistic
Variation with a Simple Electronic Instrument. Computer Music Journal, v. 36, n. 1, p. 23–41,
2012.
HADJAKOS, A.; WALOSCHEK, S. SPINE: A TUI Toolkit and Physical Computing Hybrid. (B.
Caramiaux et al., Eds.)Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...Goldsmiths, University of London, UK: Goldsmiths, University of London,
2014
HARTMANN, B. et al. d.tools: Visually Prototyping Physical UIs through Statecharts. Extended Abstracts of UIST 2005, p. 23–26, 2005.
HELMINEN, P.; AINOA, J.; MÄKINEN, S. Designing user innovation toolkits: exploring the
interrelation between solution space and module library. International Journal of Design Creativity and Innovation, n. May, p. 1–19, 2015.
HERNANDEZ, N. V.; SHAH, J. J.; SMITH, S. M. Understanding design ideation mechanisms
through multilevel aligned empirical studies. Design Studies, v. 31, n. 4, p. 382–410, 2010.
HEY, J. et al. Analogies and metaphors in creative design. International Journal of Engineering Education, v. 24, n. 2, p. 283–294, 2008.
HOLLINGER, A.; THIBODEAU, J.; WANDERLEY, M. M. An Embedded Hardware Platform for Fungible Interfaces. ICMC 2010. Anais...2010
HOOD, M. The Ethnomusicologist. New York: Kent State University Press, 1982.
HORVAT, I. Theoretical Framework for Comprehensive. International Conference on Engineering Design, n. August, p. 1–12, 2011.
HUNT, A.; WANDERLEY, M. M.; PARADIS, M. The Importance of Parameter Mapping in
Electronic Instrument Design. Journal of New Music Research, v. 32, n. 4, p. 429–440, dez.
2003.
150
HUOT, S. Designeering Interaction: A Missing Link in the Evolution of Human-Computer Interaction. [s.l.] Université Paris Sud - Paris XI, 2013.
IDEO. Human-Centered Design Toolkit: An Open-Source Toolkit to Inspire New Solutions in the Developing World. [s.l.] Authorhouse, 2011.
ISA, S. S.; LIEM, A.; STEINERT, M. the Value of Prototypes in the Early Design and Development
Process. 20th International Conference on Engineering Design (ICED 15), n. July, p. 1–8,
2015.
ISRAEL, J. H.; BÄHR, B.; EXNER, K. Perspectives on Future Prototyping—Results from an
Expert Discussion. In: Rethink! Prototyping. Cham: Springer International Publishing, 2016. p.
11–22.
JENSENIUS, A. R. et al. Musical Gestures. Musical Gestures: Sound, Movement, and Meaning, p. 12, 2010.
JOBST, B.; MEINEL, C. How Prototyping Helps to Solve Wicked Problems. In: Design Thinking Research. Cham: Springer International Publishing, 2014. p. 105–113.
JONES, J. C. Design methods. London, New York: Wiley-Interscience, 1992.
JORDÀ, S. New Musical Interfaces and New Music-making Paradigms. Proceedings of the
CHI’01 Workshop on New Interfaces for Musical Expression (NIME-01). Anais...2001
JORDÀ, S. Interactive music systems for everyone: exploring visual feedback as a way for creating more intuitive, efficient and learnable instruments. Proceedings of the Stockholm
Music Acoustics. Anais...2003
JORDÀ, S. Digital instruments and players: part I---efficiency and apprenticeship. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 59–63, 2004a.
JORDÀ, S. Digital Instruments and Players: Part II – Diversity , Freedom and Control. Proceedings of the 2004 International Computer Music Conference. Anais...2004b
JORDÀ, S. Instruments and Players: Some Thoughts on Digital Lutherie. Journal of New Music Research, v. 33, n. 3, p. 321–341, 2004c.
JORDÀ, S. Digital Lutherie. [s.l.] Universitat Pompeu Fabra, 2005a.
JORDÀ, S. Digital Lutherie Crafting musical computers for new musics’ performance and improvisation. [s.l.] Universitat Pompeu Fabra, 2005b.
JORDÀ, S. et al. The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. Proceedings of the 1st international conference on Tangible and
embedded interaction. Anais...2007
JORDÀ, S.; MEALLA, S. A Methodological Framework for Teaching , Evaluating and Informing NIME Design with a Focus on Expressiveness and Mapping. Proceedings of the
International Conference on New Interfaces for Musical Expression. Anais...Goldsmiths,
151
University of London, UK: 2014
JORDAN, B.; HENDERSON, A. Interaction Analysis: Foundations and Practice. Journal of the Learning Sciences, v. 4, n. 1, p. 39–103, 1995.
KAPTEIN, M. C.; NASS, C.; MARKOPOULOS, P. Powerful and consistent analysis of likert-type
rating scales. Proceedings of the 28th international conference on Human factors in computing systems - CHI 2010, p. 2391–2394, 2010.
KAPTELININ, V. Affordances. In: SOERGAARD, M.; DAM, R. F. (Eds.). . The Encyclopedia of Human-Computer Interaction 2nd Ed. [s.l.] Interaction Design Foundation, 2014.
KNIGHT, R. C. The KNIGHT-REVISION of HORNBOSTEL-SACHS : a new look at musical instrument classification. [s.l: s.n.].
KNORIG, A. Design Tools Design. [s.l.] FH Potsdam, 2008.
KVIFTE, T. Instruments and the Electronic Age. Oslo: Solum forlag, 1988.
KVIFTE, T.; JENSENIUS, A. R. Towards a Coherent Terminology and Model of Instrument
Description and Design. Proceedings of the conference on New interfaces for musical expression, p. 220–225, 2006.
LAW, E. L.-C. et al. Understanding, scoping and defining user experience: a survey approach. Proceedings of the SIGCHI conference on human factors in computing systems.
Anais...ACM, 2009
LEVITIN, D. J.; MCADAMS, S.; ADAMS, R. L. Control parameters for musical instruments: a
foundation for new mappings of gesture to sound. Organised Sound, v. 7, n. 2, p. 171–189, jan.
2002.
LEWIS, J. R.; SAURO, J. The factor structure of the System Usability Scale.dot.pdf. v. 1, p. 1–
10, 2009.
LIDWELL, W.; HOLDEN, K.; BUTLER, J. Universal Principles of Design. [s.l.] Rockport
Publishers, 2010.
LIM, Y. et al. Discovery-Driven Prototyping for User-Driven Creativity. p. 74–80, 2013.
LIM, Y.-K.; STOLTERMAN, E.; TENENBERG, J. The anatomy of prototypes. ACM Transactions on Computer-Human Interaction, v. 15, n. 2, p. 1–27, 1 jul. 2008.
LOWGREN, J.; STOLTERMAN, E. Thoughtful interaction design: A design perspective on information technology. [s.l.] The MIT press, 2004.
LUCIANI, A. Ergotic / epistemic / semiotic functions. In: Enaction and enactive interfaces: a handbook of terms, Enactive System Books. [s.l: s.n.]. p. 96–97.
MACHOVER, T. Hyperinstruments: A composer’s approach to the evolution of intelligent musical
instruments. Organized Sound, p. 67–76, 1991.
MAESTRACCI, B.; FRECHIN, J.; PETREVSKI, U. Modular Musical Objects Towards
152
Embodied Control Of Digital Music. Proceedings of the International Conference on Tangible,
Embedded and Embodied Interaction (TEI). Anais...2011
MAGNUSSON, T. Designing constraints: composing and performing with digital musical systems.
Computer Music Journal, v. 34, n. 4, p. 62–74, 2010.
MAGNUSSON, T.; MENDIETA, E. H. The Acoustic, the Digital and the Body : A Survey on
Musical Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 94–99, 2007a.
MAGNUSSON, T.; MENDIETA, E. H. E. The acoustic, the digital and the body: A survey on
musical instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 94–99, 2007b.
MALLOCH, J. et al. Towards a New Conceptual Framework for Digital Musical Instruments.
Proceedings of the 9th International Conference on Digital Audio Effects. Anais...2006
MALLOCH, J. A Consort of Gestual Musical Controllers: Design, Construction and Performance. [s.l.] McGill University, 2008.
MALLOCH, J.; SINCLAIR, S.; WANDERLEY, M. M. Distributed tools for interactive design of
heterogeneous signal networks. Multimedia Tools and Applications, p. 1–25, 2014.
MALONEY, T. M. The family of wood composite materials. Forest Products Journal, v. 46, n.
2, p. 18, 1996.
MARQUEZ-BORBON, A. et al. Designing Digital Musical Interactions in Experimental Contexts. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...Oslo, Norway: 2011
MARTIN, R. C. Agile Software Development: Principles, Patterns, and Practices. [s.l.]
Prentice Hall PTR, 2003.
MATHEWS, M. Radio Baton, 2005.
MAURYA, A. Running Lean: Iterate from Plan A to a Plan That Works. [s.l.] O’Reilly Media,
2012.
MEDEIROS, C.; WANDERLEY, M. M. A Comprehensive Review of Sensors and Instrumentation
Methods in Devices for Musical Expression. Sensors, v. 14, n. 8, p. 13556–13591, 25 jul. 2014.
MEDEIROS, R. et al. Challenges in Designing New Interfaces for Musical Expression. In: Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience. [s.l.] Springer, 2014. p. 643–652.
MERRIAM-WEBSTER. Merriam-Webster’s collegiate dictionary. [s.l.] Merriam-Webster,
2004.
MIMO CONSORTIUM. Revision of the Hornbostel-Sachs Classification of Musical Instruments by the MIMO Consortium. [s.l: s.n.].
153
MIRANDA, E. R. Computer Sound Synthesis for the Electronic Musician. [s.l.] Butterworth-
Heinemann, 1998.
MIRANDA, E. R.; WANDERLEY, M. M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Middleton: A-R Editions, 2006.
MONTAG, M. et al. A Low-Cost , Low-Latency Multi-Touch Table with Haptic Feedback for Musical Applications. Proceedings of the International Conference on New Interfaces for
Musical Expression. Anais...Oslo, Norway: 2011
MOREY, R. D. Confidence Intervals from Normalized Data: A correction to Cousineau ( 2005 ).
Tutorials in Quantitative Methods for Psychology, v. 4, n. 2, p. 61–64, 2008.
MORREALE, F.; ANGELI, A. DE; O’MODHRAIN, S. Musical Interface Design: An Experience-oriented Framework. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...Goldsmiths, University of London, UK: 2014
MULDER, A. The I-Cube System: moving towards sensor technology for artists.
Proceedings of the ISEA. Anais...1995
MURRAY-BROWNE, T. et al. The medium is the message : Composing instruments and performing mappings. Proceedings of the International Conference on New Interfaces for
Musical Expression. Anais...Oslo, Norway: 2011
MUSER, S. Gestures in Human-Computer-Interaction. [s.l: s.n.].
NAKAMARU, S. Tangible Modeling Methods for Faster Rapid Prototyping. p. 518–523, 2016.
NAKAMURA, J.; CSIKSZENTMIHALYI, M. The concept of flow. In: Flow and the foundations of positive psychology. [s.l.] Springer, 2014. p. 239–263.
NEVES, A. Design Thinking Canvas. [s.l: s.n.].
NEWTON-DUNN, H.; NAKANO, H.; GIBSON, J. Block Jam: A Tangible Interface for Interactive Music. Proceedings of the 2003 Conference on New Interfaces for Musical
Expression (NIME-03). Anais...Montreal, Canada: dez. 2003
NIELS, B.; GELINECK, S.; SERAFIN, S. PHYSMISM : A control interface for creative exploration of physical models . Proceedings of the 2007 Conference on New Interfaces for
Musical Expression (NIME07). Anais...2007
NISHIBORI, Y.; IWAI, T. Tenori-on. Proceedings of the 2006 International Conference on New
Interfaces for Musical Expression (NIME06). Anais...Paris, France: 2006
NORMAN, D. A. The Design of Everyday Things: Revised and Expanded Edition. [s.l: s.n.].
NORT, D. VAN; WANDERLEY, M. M. The LoM Mapping Toolbox for Max/MSP/Jitter. Proceedings of the International Computer Music Conference. Anais...2006
O’HARA, D. Skeuomorphology and Quotation. In: ROUSSEL, M.; BORKENHAGEN, C. (Eds.). .
Creativity of Finding: Figurations of the Quotation. [s.l.] Wilhelm Fink Verlag, 2012. v. 2p.
154
281–293.
O’MODHRAIN, S. A framework for the evaluation of digital musical instruments. Computer Music Journal, v. 35, n. 1, p. 28–42, 2011.
O’MODHRAIN, S.; CHAFE, C. Incorporating haptic feedback into interfaces for music applications. Proceedings of the International Symposium on Robotics with Applications, World
Automation Conference. Anais...2000
OORE, S. Learning Advanced Skills on New Instruments (or practising scales and arpeggios on your NIME). Proceedings of the 2005 International Conference on New Interfaces
for Musical Expression (NIME05). Anais...Vancouver, BC, Canada: 2005
OSCULATOR. OSCulator.
OWEN, C. Considering Design Fundamentally. Design Processes Newsletter. Anais...1993
PAHL, G. et al. Engineering design: a Systematic Approach. [s.l: s.n.].
PAINE, G. Towards Unified Design Guidelines for New Interfaces for Musical Expression.
Organised Sound, v. 14, n. 2, p. 142, jun. 2009.
PAINE, G. New Musical Instrument Design Considerations. MultiMedia, IEEE, p. 76–84, 2013.
PAINE, G. Interaction as Material: The techno-somatic dimension. Organised Sound, v. 20, n.
1, p. 82–89, 2015.
PALACIO-QUINTIN, C. The Hyper-Flute. Proceedings of the 2003 Conference on New
Interfaces for Musical Expression (NIME-03). Anais...Montreal, Canada: 2003
PASSERA, S.; KAERKKAEINEN, H.; MAILA, R. When, how, why prototyping? A practical
framework for service development. n. Brown 2008, p. 1–16, 2012.
POTIDIS, S.; SPYROU, T. Spyractable: A tangible user interface modular synthesizer. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 8511 LNCS, n. PART 2, p. 600–611, 2014.
PREECE, J. et al. Interaction design: Beyond human-computer Interaction. 4th. ed. [s.l.]
John Wiley, 2015.
PUCCIO, G. J.; CABRA, J. F. Idea generation and idea evaluation: Cognitive skills and deliberate practices. [s.l.] Elsevier Inc., 2012.
RAMALHO, L. F. DE O. Herança Gestual Aplicada Em Design de Novos Artefatos de Interação Complexa. [s.l.] Universidade Federal de Pernambuco, 2017.
REINIG, B. A.; BRIGGS, R. O. On the relationship between idea-quantity and idea-quality during
ideation. Group Decision and Negotiation, v. 17, n. 5, p. 403–420, 2008.
RESNICK, M.; SILVERMAN, B. Some Reflections on Designing Construction Kits for Kids.
Proceeding of the 2005 Conference on Interaction Design and Children (IDC ’05), p. 117–
122, 2005.
155
RITCHEY, T. General Morphological Analysis A general method for non-quantified modelling.
16th EURO Conference on Operational Analysis, p. 1–10, 1998.
ROECK, D. D. E. et al. Beyond Static : Ideation Using Interactive Prototyping Toolkits.
International Conference On Engineering And Product Design Education. Anais...2013
ROVAN, J. B. et al. Instrumental gestural mapping strategies as expressivity determinants in computer music performance. Proceedings of Kansei-The Technology of Emotion
Workshop. Anais...Citeseer, 1997
RYAN, J. Some remarks on musical instrument design at STEIM. Contemporary Music Review,
v. 6, n. 1, p. 3–17, 1991.
SACHS, C. The History of Musical Instruments. [s.l.] W. W. Norton & Company, Inc., 1940.
SADLER, J. et al. Bloctopus: A Novice Modular Sensor System for Playful Prototyping.
Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI ’14, p. 347–354, 2015.
SADLER, J. The Anatomy of Creative Computing: Enabling Novices to Prototype Smart Devices. [s.l.] Stanford University, 2016.
SADLER, J. et al. Building Blocks of the Maker Movement: Modularity Enhances Creative
Confidence During Prototyping. In: PLATTNER, H.; MEINEL, C.; LEIFER, L. (Eds.). . Design Thinking Research. [s.l.] Springer International Publishing, 2016a. p. 141–154.
SADLER, J. et al. Can Anyone Make a Smart Device?: Evaluating the Usability of a Prototyping
Toolkit for Creative Computing. In: AL., H. P. ET (Ed.). . Design Thinking Research, Understanding Innovation. [s.l.] Springer International Publishing Switzerland 2016, 2016b. p.
147–160.
SALDAÑA, J. The Coding Manual for Qualitative Researchers. [s.l: s.n.].
SANDERS, E. B.-N. Prototyping For The Design Spaces Of The Future. Prototype: Design and Craft in the 21st Century, p. 1–8, 2013.
SANDERS, E. B.-N.; STAPPERS, P. J. From Designing to Co-Designing to Collective Dreaming:
Three Slices in Time. Interactions, p. 24–33, 2014.
SCHLOSS, W. Recent advances in the coupling of the language Max with the Mathews/Boie Radio Drum. Proceedings of the International Computer Music Conference. Anais...1990
SCHMEDER, A.; FREED, A. uosc: The open sound control reference platform for embedded devices. Proceedings of the 2008 Conference on New Interfaces for Musical Expression
(NIME08). Anais...Citeseer, 2008
SHAH, J.; KULKARNI, S.; VARGAS-HERNANDEZ, N. Evaluation of Idea Generation Methods
for Conceptual Design: Effectiveness Metrics and Design of Experiments. Journal of Mechanical Design, v. 122, n. 4, p. 377–384, 2000.
156
SHESKIN, D. J. Handbook of Parametric and Nonparametric Statistical Procedures: Third Edition. [s.l: s.n.].
SMITH, G. F. Idea-Generation Techniques : A Formulary of Active Ingredients. The Journal of Creative Behavior, v. 32, n. 2, p. 107–134, 1998.
STEIM. junXion. Disponível em: <http://steim.org/product/junxion/>.
STEINER, H. Building your own instrument with Pd. Proceedings of the 1st International Pd
Conference. Anais...2005
STOWELL, D. et al. Evaluation of live human–computer music-making: Quantitative and
qualitative approaches. International Journal of Human-Computer Studies, v. 67, n. 11, p.
960–975, nov. 2009.
SYLLEROS, A.; DE LA CUADRA, P.; CÁDIZ, R. Designing a Musical Instrument: Enlivening
Theory Through Practice-Based Research. Design Issues, v. 30, n. 2, p. 83–96, abr. 2014.
TANAKA, A. et al. A Survey and Thematic Analysis Approach as Input to the Design of Mobile Music GUIs. Proceedings of the International Conference on New Interfaces for Musical
Expression. Anais...University of Michigan, Ann Arbor: 2012
TANAKA, A.; ALTAVILLA, A.; SPOWAGE, N. Gestural Musical Affordances. Proc. of Sound and Music Computing Conference (SMC), p. 318–325, 2012.
TEBOUL, E. The Transgressive Practices of Silicon Luthiers. In: Guide to Unconventional Computing for Music. Cham: Springer International Publishing, 2017. p. 85–120.
TORRE, G.; ANDERSEN, K.; BALDÉ, F. The Hands: The Making of a Digital Musical Instrument.
Computer Music Journal, v. 40, n. 2, p. 22–34, jun. 2016.
TRUEMAN, D.; DUBOIS, R. L. Percolate: a collection of synthesis, signal processing, and image processing objects. Disponível em: <https://github.com/Cycling74/percolate>. Acesso
em: 21 fev. 2017.
UDELL, C.; SAIN, J. P. eMersion: Sensor-controlled Electronic Music Modules & Digital Data Workstation. (B. Caramiaux et al., Eds.)Proceedings of the International Conference on
New Interfaces for Musical Expression. Anais...Goldsmiths, University of London, UK:
Goldsmiths, University of London, 2014
VALAMANESH, R.; SHIN, D. Tangible ideation: How digital fabrication acts as a catalyst in the
early steps of product development. Proceedings of the 15th International Conference on Engineering and Product Design Education: Design Education - Growing Our Future, EPDE 2013, n. September, p. 635–640, 2013.
VASCONCELOS, L. A. et al. Investigating real-world design fixation using morphological analysis. Design Creativity Workshop. Anais...Chicago: Northwestern University, Evanston,
2016
VASCONCELOS, L. A.; CRILLY, N. Inspiration and fixation: Questions, methods, findings, and
157
challenges. Design Studies, v. 42, p. 1–32, 2016.
VASHLISHAN, M. J. The Akai Electric Wind Instrument (EWI4000s): A Technical and Expressive Method. [s.l.] University of Miami, 2011.
VERTEGAAL, R.; UNGVARY, T.; KIESLINGER, M. Towards a musician’s cockpit: Transducers, feedback and musical function. Proceedings of the International Computer
Music Conference. Anais...1996
VILLAR, N.; LINDSAY, A.; GELLERSEN, H. Pin & Play & Perform: a rearrangeable interface for musical composition and performance. Proceedings of the 2005 International Conference
on New Interfaces for Musical Expression (NIME05). Anais...Vancouver, BC, Canada: 2005
VISWANATHAN, V. et al. Prototyping: A key skill for innovation and life-long learning.
Proceedings - Frontiers in Education Conference, FIE, v. 2015–Febru, n. February, 2015.
VON HIPPEL, E. PERSPECTIVE: User toolkits for innovation. Journal of Product Innovation Management, v. 18, n. 4, p. 247–257, 2001.
VON THIENEN, J.; MEINEL, C.; NICOLAI, C. How Design Thinking Tools Help To Solve Wicked
Problems. In: Design Thinking Research. Cham: Springer International Publishing, 2014. v.
36p. 97–102.
WALLIS, I. et al. Amateur Musicians, Long-Term Engagement, and HCI. In: HOLLAND, S. et al.
(Eds.). . Music and Human-Computer Interaction. [s.l.] Springer London, 2013. p. 49–66.
WALTER-HERRMANN, J.; BÜCHING, C. FabLab: Of Machines, Makers and Inventors. [s.l.]
transcript Verlag, 2014.
WANDERLEY, M. M. Instrumentos Musicais Digitais: Gestos, Sensores e Interfaces. In: Em Busca da Mente Musical. [s.l.] Editora da Universidade Federal do Paraná, 2006. v. 60.
WANDERLEY, M. M.; ORIO, N. Evaluation of Input Devices for Musical Expression: Borrowing
Tools from HCI. Computer Music Journal, v. 26, n. 3, p. 62–76, 2002.
WARD, N.; TORRE, G. Constraining Movement as a Basis for DMI Design and Performance. Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...Goldsmiths, University of London, UK: 2014
WARFEL, T. Z. Prototyping: A Practitioner’s Guide. [s.l.] Rosenfeld Media, 2009.
WASSERMAN A., E.; CULLEN, P. Evolution of the Violin: The Law of Effect in Action. Journal of Experimental Psychology: Animal Learning and Cognition, 2015.
WESSEL, D.; WRIGHT, M. Problems and Prospects for Intimate Musical Control of Computers. Proceedings of the CHI’01 Workshop on New Interfaces for Musical Expression
(NIME-01). Anais...2001
WIETHOFF, A. Prototyping Tools for Hybrid Interactions. [s.l.] Ludwig-Maximilians-Universität
München, 2012.
158
WOOD, L. E. Semi-structured interviewing for user-centered design. interactions, v. 4, n. 2, p.
48–61, 1997.
WRIGHT, M.; FREED, A.; MOMENI, A. OpenSound Control: State of the Art 2003.
Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03).
Anais...Montreal, Canada: 2003
XAMBÓ, A. Tabletop Tangible Interfaces for Music Performance: Design and Evaluation.
[s.l.] The Open University, 2015.
YOUMANS, R. J. The effects of physical prototyping and group work on the reduction of design
fixation. Design Studies, v. 32, n. 2, p. 115–138, 2011.
ZAPPI, V.; MCPHERSON, A. Dimensionality and Appropriation in Digital Musical Instrument Design. Proceedings of the International Conference on New Interfaces for Musical Expression.
Anais...Goldsmiths, University of London, UK: 2014
ZWICKY, F. The Morphological Approach to Discovery, Invention, Research and Construction.
New Methods of Thought and Procedure, p. 273–297, 1967.
159
APPENDIX A - RELATED PROJECTS Tool Reference
Arduino http://arduino.cc
Bareconductive Touch Board https://www.bareconductive.com/shop/touch-board/
BeagleBoard https://beagleboard.org/
BELA http://bela.io/
bitalino http://www.bitalino.com/
Block Jam (NEWTON-DUNN; NAKANO; GIBSON, 2003)
Bloctopus (SADLER et al., 2015)
d.tools (HARTMANN et al., 2005)
eMersion (UDELL; SAIN, 2014)
fungible interfaces (HOLLINGER; THIBODEAU; WANDERLEY, 2010)
i-cubeX http://infusionsystems.com/catalog/
Interface Z http://www.interface-z.fr/
iStuff (BALLAGAS et al., 2003)
Joué http://www.play-joue.com/
Keys https://igg.me/at/keys/x
LightPad https://roli.com/products/blocks
littleBits Korg https://littlebits.cc/kits/synth-kit
Makey Makey http://www.makeymakey.com
mbed https://www.mbed.com
microduino mCookies https://www.microduino.cc/
Mine Modular Controller http://special-waves.com/
MODI http://www.luxrobo.com/
Modulares Interface https://vimeo.com/108885687
Modulome (BARRACLOUGH; MURPHY; KAPUR, 2014)
mogees http://www.mogees.co.uk/
160
Molecule Synth http://moleculesynth.com/
Nascent Objects http://www.nascentobjects.com/
Neo http://www.lolagielen.nl/neo.html
OpenDeck https://github.com/paradajz/OpenDeck
oplab https://teenage.engineering/products/oplab
Ototo http://www.ototo.fm
Palette https://palettegear.com/
Phidgets http://www.phidgets.com/
PHOXES (GELINECK; SERAFIN, 2010b)
PHYSMISM (NIELS; GELINECK; SERAFIN, 2007)
Pin&Play&Perform (VILLAR; LINDSAY; GELLERSEN, 2005)
PowderBox http://yoshihito-nakanishi.com/works/device/powder-box/
Pulse Controller http://www.tetmusic.com/
Raspberry PI https://www.raspberrypi.org/
Reactable http://reactable.com/
SAM https://www.samlabs.com/
Satellite CCRMA (BERDAHL; JU, 2011)
Sifteo http://www.sifteo.com
Sound Clippys Platform http://modular-muse.com/digital-music-instruments/
SPINE (HADJAKOS; WALOSCHEK, 2014)
Spyractable (POTIDIS; SPYROU, 2014)
TeaBox (ALLISON; PLACE, 2005)
Tessel https://tessel.io/
TinyDuino https://tiny-circuits.com/tinyduino_overview
UI Prototyping Kit (FEIL; TUNG, 2013)
Video-Organ (BONGERS; HARRIS, 2002)
x-OSC http://x-io.co.uk/