publications

Full Conference Papers

2024

1. AM

   Timbre Tools: Ethnographic Perspectives on Timbre and Sonic Cultures in Hackathon Designs

   Saitis, Charalampos, Sette, Bleiz Macsen Del, Shier, Jordie, Tian, Haokun, Zheng, Shuoyang, Skach, Sophie, Reed, Courtney N., and Ford, Corey

   In Proceedings of the 2024 ACM Audio Mostly Conference 2024

   Abs PDF

   Timbre is a nuanced yet abstractly defined concept. Its inherently subjective qualities make it challenging to design and work with. In this paper, we propose to explore the conceptualisation and negotiation of timbre within the creative practice of timbre tool makers. To this end, we hosted a hackathon event and performed an ethnographic study to explore how participants engaged with the notion of timbre and how their conception of timbre was shaped through social interactions and technological encounters. We present individual descriptions of each team’s design process and reflect on our data to identify commonalities in the ways that timbre is understood and informed by sound technologies and their surrounding sonic cultures, e.g., by relating concepts of timbre to metaphors. We further current understanding by offering novel interdisciplinary and multimodal insights into understandings of timbre.

2. C&C

   Reflection Across AI-based Music Composition

   Ford, Corey, Noel-Hirst, Ashley, Cardinale, Sara, Loth, Jackson, Sarmento, Pedro, Wilson, Elizabeth, Wolstanholme, Lewis, Worrall, Kyle, and Bryan-Kinns, Nick

   In Proceedings of the 16th Conference on Creativity & Cognition 2024

   Abs PDF

   Reflection is fundamental to creative practice. However, the plurality of ways in which people reflect when using AI Generated Content (AIGC) is underexplored. This paper takes AI-based music composition as a case study to explore how artist-researcher composers reflected when integrating AIGC into their music composition process. The AI tools explored range from Markov Chains for music generation to Variational Auto-Encoders for modifying timbre. We used a novel method where our composers would pause and reflect back on screenshots of their composing after every hour, using this documentation to write first-person accounts showcasing their subjective viewpoints on their experience. We triangulate the first-person accounts with interviews and questionnaire measures to contribute descriptions on how the composers reflected. For example, we found that many composers reflect on future directions in which to take their music whilst curating AIGC. Our findings contribute to supporting future explorations on reflection in creative HCI contexts.

3. C&C

   Using Incongruous Genres to Explore Music Making with AI Generated Content

   Bryan-Kinns, Nick, Noel-Hirst, Ashley, and Ford, Corey

   In Proceedings of the 16th Conference on Creativity & Cognition 2024

   Abs PDF

   Deep learning generative AI models trained on huge datasets are capable of producing complex and high quality music. However, there are few studies of how AI Generated Content (AIGC) is actually used or appropriated in creative practice. We present two first-person accounts by musician-researchers of explorations of an interactive generative AI system trained on Irish Folk music. The AI is intentionally used by musicians from incongruous genres of Punk and Glitch to explore questions of how the model is appropriated into creative practice and how it changes creative practice when used outside of its intended genre. Reflections on the first-person accounts highlight issues of control, ambiguity, trust, and filtering AIGC. The accounts also highlight the role of AI as an audience and critic and how the musicians’ practice changed in response to the AIGC. We suggest that our incongruous approach may help to foreground the creative work and frictions in human-AI creative practice.

2023

1. CHI

   Towards a Reflection in Creative Experience Questionnaire

   Ford, Corey, and Bryan-Kinns, Nick

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems Apr 2023

   Abs PDF

   Reflection is underexplored in Creativity Support Tool (CST) research, partly due to its ambiguous nature. We suggest that researchers could benefit from a measure of a CST’s capacity to support reflection. To this end, we detail the first stages of development of the Reflection in Creative Experience Questionnaire (RiCE) – a lightweight questionnaire for differentiating between creative user experiences which exhibit more or less moments of reflection. We develop RiCE through i) an expert review of questionnaire items (n=10) and ii) an exploratory factor analysis (n=300) of the reviewed items. We also present a user study testing RiCE (n=58) across two time points (one week apart) with novel interfaces designed for creative writing and music making. Although we do not confirm validity, we identify four factors for RiCE which we suggest are interpretable in a conceptually meaningful way. Our formative studies contribute towards supporting future explorations on reflection with CSTs.

2022

1. C&C

   Identifying Engagement in Children’s Interaction Whilst Composing Digital Music at Home

   Ford, Corey, and Bryan-Kinns, Nick

   In Proceedings of the ACM SIGCHI Conference on Creativity and Cognition Jun 2022

   Abs PDF

   Identifying points of engagement from a person’s interaction with computers could be used to assess their experience and to adapt user interfaces in real-time. However, it is difficult to identify points of engagement unobtrusively; HCI studies typically use retrospective protocols or rely on cumbersome sensors for real-time analysis. We present a case study on how children compose digital music at home in which we remotely identify points of engagement from patterns of interaction with a musical interface. A mixed-methods approach is contributed in which video recordings of children’s interactions whilst composing are labelled for engagement and linked to i) interaction logs from the interface to identify indicators of engagement in interaction, and ii) interview data gathered using a remote video-cued recall technique to understand the experiential qualities of engaging interactions directly from users. We conclude by speculating on how the suggested indicators of engagement inform the design of adaptive music systems.

2021

1. NIME

   Creativity in Children’s Digital Music Composition

   Ford, Corey, Bryan-Kinns, Nick, and Nash, Chris

   In Proceedings of the International Conference on New Interfaces for Musical Expression Jun 2021

   Abs PDF

   Composing is a neglected area of music education. To increase participation, many technologies provide open-ended interfaces to motivate child autodidactic use, drawing influence from Papert’s LOGO philosophy to support children’s learning through play. This paper presents a case study examining which interactions with Codetta, a LOGO-inspired, block-based music platform, supports children’s creativity in music composition. Interaction logs were collected from 20 children and correlated against socially-validated creativity scores. To conclude, we recommend that the transition between low-level edits and high-level processes should be carefully scaffolded.

2020

1. NIME

   An Iterative Design ‘by proxy’ Method for Developing Educational Music Interfaces

   Ford, Corey, and Nash, Chris

   In Proceedings of the International Conference on New Interfaces for Musical Expression Jul 2020

   Abs PDF

   Iterative design methods involving children and educators are difficult to conduct, given both the ethical implications and time commitments understandably required. The qualitative design process presented here recruits introductory teacher training students, towards discovering useful design insights relevant to music education technologies “by proxy”. Therefore, some of the barriers present in child-computer interaction research are avoided. As an example, the method is applied to the creation of a block-based music notation system, named Codetta. Building upon successful educational technologies that intersect both music and computer programming, Codetta seeks to enable child composition, whilst aiding generalist educator’s confidence in teaching music.

Book Chapters

2024

1. T&F

   Explainable AI and Music

   Bryan-Kinns, Nick, Banar, Berker,  Ford, Corey, Reed, Courtney N, Zhang, Yixiao, and Armitage, Jack

   In Proceedings of the International Conference on New Interfaces for Musical Expression Jul 2024

   Abs PDF

   The field of eXplainable Artificial Intelligence (XAI) has become a hot topic examining how machine learning models such as neural nets and deep learning techniques can be made more understandable to humans. However, there is very little research on XAI for the arts. This chapter explores what XAI might mean for AI and art creation by exploring the potential of XAI for music generation. One hundred AI and music papers are reviewed to illustrate how AI models are being explained, or more often not explained, and to suggest some ways in which we might design XAI systems to better help humans to get an understanding of what an AI model is doing when it generates music. Then the chapter demonstrates how a latent space model for music generation can be made more explainable by extending the MeasureVAE architecture to include explainable attributes in combination with offering real-time music generation. The chapter concludes with four key challenges for XAI for music and the arts more generally: i) the nature of explanation; ii) the effect of AI models, features, and training sets on explanation; iii) user centred design of XAI; iv) Interaction Design of explainable interfaces.

Workshop & Short Papers

2024

1. C&C

   Explainable AI for the Arts 2 (XAIxArts2)

   Bryan-Kinns, Nick,  Ford, Corey, Zheng, Shuoyang, Kennedy, Helen, Chamberlain, Alan, Lewis, Makayla, Hemment, Drew, Li, Zijin, Wu, Qiong, Xiao, Lanxi, Xia, Gus, Rezwana, Jeba, Clemens, Michael, and Vigliensoni, Gabriel

   In Proceedings of the 16th Conference on Creativity & Cognition Jul 2024

   Abs PDF

   This second workshop on explainable AI for the Arts (XAIxArts) brings together a community of researchers and creative practitioners in Human-Computer Interaction (HCI), Interaction Design, AI, explainable AI (XAI), and Digital Arts to explore the role of XAI for the Arts. XAI is a core concern of Human-Centred AI and relies heavily on HCI techniques to explore how to make complex and difficult to understand AI models more understandable to people. Our first workshop explored the landscape of XAIxArts and identified emergent themes. To move the discourse on XAIxArts forward and to contribute to Human-Centred AI more broadly this workshop will: i) bring researchers together to expand the XAIxArts community; ii) collect and critically reflect on current and emerging XAIxArts practice; iii) co-develop a manifesto for XAIxArts; iv) co-develop a proposal for an edited book on XAIxArts; v) engage with the wider discourse on Human-Centred AI.

2. alt.chi

   Traveling Arts x HCI Sketchbook: Exploring the Intersection Between Artistic Expression and Human-Computer Interaction

   Lewis, Makayla, Sturdee, Miriam, Lengyel, Denise, Toselli, Mauro, Miers, John, Owen, Violet, Davis, Josh Urban, Gaudl, Swen E, Xiao, Lanxi, Priego, Ernesto, Snooks, Kim, Turmo Vidal, Laia, Blevis, Eli, Privato, Nicola, Piedade, Patricia,  Ford, Corey, Bryan-Kinns, Nick, Severes, Beatriz, Kaipainen, Kirsikka, Claisse, Caroline, Huq, Raksanda Mehnaz, Eladhari, Mirjam Palosaari, Troisi, Anna, Henriques, Ana O, Grek, Ar, Mcmurchy, Gareth, Lc, Ray, Nabil, Sara, Jardine, Jacinta, Collins, Robert, Vlasov, Andrey, Knight, Yana, Cremaschi, Michele, Carderelli-Gronau, Silvia, Núñez-Pacheco, Claudia, Reyes-Cruz, Gisela, and Riviere, Jean-Philippe

   In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems Jul 2024

   Abs PDF

   When thinking of arts in HCI, one might be tempted to keep one’s eyes focused on prominent realms such as sketching for UX Design and design probes from participants. A closer look shows that practices go beyond this, involving a variety of arts-based expressions by researchers, the researched and third parties, e.g. graphic facilitators. Inspired by Toselli’s Sketchnote Army Travelling Sketchbook, researchers and artists contributed to a ’Travelling Sketchbook for Arts in HCI’, showcasing their arts-based practice in HCI. The resulting sketchbook explores the intersection between HCI and artistic expression, illuminating what it means to use art in HCI. It shows the breadth of Arts in HCI, illustrating the many fruitful possibilities for extending existing research and dissemination methods in HCI. It also calls into question current practices, which often do not recognise the significance of artist attribution, and, in turn, advocates for equal authorship between principal researchers and contributing artists.

2023

1. CHIME

   Three Open Questions for the Design of AI for Music Composition

   Ford, Corey, Cardinale, Sara, Bryan-Kinns, Nick, and Wiggins, Geraint

   In CHIME One Day Workshop 2023 Dec 2023

   Abs PDF

   This paper suggests three open questions for designing AI for music composition by reflecting on interviews with expert and novice composers. Our questions consider the role of AI as a mediator, the importance of a composer’s originality, and AI for building intuition.

2. C&C

   Explainable AI for the Arts: XAIxArts

   Bryan-Kinns, Nick,  Ford, Corey, Chamberlain, Alan, Benford, Steven David, Kennedy, Helen, Li, Zijin, Qiong, Wu, Xia, Gus G., and Rezwana, Jeba

   In Proceedings of the 15th Conference on Creativity and Cognition Dec 2023

   Abs PDF

   This first workshop on explainable AI for the Arts (XAIxArts) brings together a community of researchers and creative practitioners in Human-Computer Interaction (HCI), Interaction Design, AI, explainable AI (XAI), and Digital Arts to explore the role of XAI for the Arts. XAI is a core concern of Human-Centred AI and relies heavily on HCI techniques to explore how complex and difficult to understand AI models such as deep learning techniques can be made more understandable to people. However, XAI research has primarily focused on work-oriented and task-oriented explanations of AI and there has been little research on XAI for creative domains such as the Arts. This workshop will: i) build an XAIxArts research community; ii) map out the current and future possible landscapes of XAIxArts; iii) critically reflect on the potential of XAI for the Arts, forming the basis for an edited book on XAIxArts and an international network of researchers.

3. Digital Skills

   On the Role of Reflection and Digital Tool Design for Creative Practioners

   Ford, Corey, and Bryan-Kinns, Nick

   In 1st Workshop on Digital Skills for the Creative Practitioner: Supporting Informal Learning of Technologies for Creativity Apr 2023

   Abs PDF

   This position paper suggests that Creativity Support Tools (CSTs) could be designed for reflection to foster self-learning in creative user experiences. The author’s suggest that a state where people continually apply reflection and conscious processing is likely con- ducive to supporting creative practitioners use of and self-learning of digital tools, contrasting theories which encourage passive in- teraction which forgoes self-awareness cf. flow theory. To this end, two case studies from the first author’s PhD research are introduced: i) a questionnaire to differentiate between creative user experiences with more or less moments of reflection, and ii) a summary of the iterative design of a block-based CST to encourage moments of reflection in people’s generative music composition with AI. This research is then discussed in the context of the workshop themes.

2022

1. GenAI & HCI

   Speculating on Reflection and People’s Music Co-Creation with AI

   Ford, Corey, and Bryan-Kinns, Nick

   In Workshop on Generative AI and HCI at CHI 2022 (GenAI & HCI @ CHI 2022) Mar 2022

   Abs PDF

   Recent advances in generative Artificial Intelligence (AI) and Human-Computer Interaction (HCI) have led to a design trend of user interfaces supporting novices in co-creating with AI. Typically, these tools aim to extend novices’ abilities and democratise the more difficult aspects of creative practice. We suggest that designing for reflection might be an interesting alternative avenue to explore in novice’s creative co-creation with generative AI. We contribute a systematic literature review of 122 creative AI co-creators – mostly for music – with respect to HCI theories on reflection. We then explore findings from our literature review within the domain of music, suggesting an interface design for an interactive tool based on the current trend in music AI to expose the latent space of variational autoencoders.

2021

1. XAI @ NeurIPS

   Exploring XAI for the Arts: Explaining Latent Space in Generative Music

   Bryan-Kinns, Nick, Banar, Berker,  Ford, Corey, Reed, Courtney N., Zhang, Yixiao, Colton, Simon, and Armitage, Jack

   In 1st Workshop on eXplainable AI Approaches for Debugging and Diagnosis (XAI4Debugging@NeurIPS2021) Dec 2021

   Abs PDF

   Explainable AI has the potential to support more interactive and fluid co-creative AI systems which can creatively collaborate with people. To do this, creative AI models need to be amenable to debugging by offering eXplainable AI (XAI) features which are inspectable, understandable, and modifiable. However, currently there is very little XAI for the arts. In this work, we demonstrate how a latent variable model for music generation can be made more explainable; specifically we extend MeasureVAE which generates measures of music. We increase the explainability of the model by: i) using latent space regularisation to force some specific dimensions of the latent space to map to meaningful musical attributes, ii) providing a user interface feedback loop to allow people to adjust dimensions of the latent space and observe the results of these changes in real-time, iii) providing a visualisation of the musical attributes in the latent space to help people understand and predict the effect of changes to latent space dimensions. We suggest that in doing so we bridge the gap between the latent space and the generated musical outcomes in a meaningful way which makes the model and its outputs more explainable and more debuggable. The code repository can be found at: https://github.com/bbanar2/Exploring_XAI_in_GenMus_via_LSR