Cantinas

Cantinas were a series of monthly casual video chats, organized by the on-the-fly research group. Each session was hosted by one or more members of the community revolving around a different question. Rather than focusing on tools and specific programming languages, these meetups provided a space to discuss aesthetics, meta levels and how we relate to live coding - both as artists and as audience. Moreover, specific subjects, concerns, perspectives and questions ranging from virtuality to machine learning were discussed between peers. The series of cantinas carried out set the pace of the research line of the project. At the following link you will find a report "on-the-fly-cantina" of some Cantinas described by the members of the research group, named: Iván Paz, Anne Veinberg, Patrick Borgeat and Luka Frelih.

Introduction

on-the-fly is a Creative Europe co-funded project initiated by Hangar in Barcelona, in cooperation with Creative Coding Utrecht, Ljudmila Art and Science Laboratory, Ljubljana, and ZKM | Center for Art and Media Karlsruhe, which aims to support and foster the exchange of the different practices among live coding communities. In this research space we discuss, explore, identify, reflect on and imagine creative directions in the live coding practice. Live coding activities have many layers and composer-programmers have spent extensive hours designing, understanding, or extending numerous languages or libraries before performing with them. In addition, reflecting on live coding nature and possibilities contextualizes and restricts the possible paths to follow. This is an essential process as the possibilities would otherwise be infinite. With this in mind, the on-the-fly research group holds monthly casual video meetups for the live coding community under the title on-the-fly cantina. Each session follows a distinct central question and is hosted by a member of the research group or another member from the live coding community. Rather than focusing on tools and specific programming languages, these meetups have provided a space to discuss aesthetics, meta levels and how we relate to live coding - both as artists and as audience. In this paper we report on four of these sessions that touched on different live coding practices, activities, concepts and artistic questions. The topics included in this paper represent an incomplete live coding panorama and the aim is not to give an elaborate overview or thorough discussion but to present a compilation of opinions and ideas exchanged in the video chat sessions. Nevertheless these reports might serve as a pool of ideas that can be investigated further in a more thorough and scientific way or can just be used for personal inspiration. The central questions of the sessions reported on in this paper are: “ Now that we have gathered - what shall we discuss?”, “How alive is a live-streamed Algorave”, and ˝Live Coding Machine Learning˝.

Now that we have gathered - what shall we discuss?

True to the meeting’s theme, the opening session of the on-the-fly cantina made space for participants to share their thoughts, ideas, struggles and interests in live coding. Following a brief introduction by those present, a number of themes and questions were brought up. These can be subdivided into a few main areas. 1) When is live coding, live coding and the relevance of the live coding manifesto. 2) the music and language trends in live coding. 3) the performance practice of live coding. 4) Other topics.

2.1 When is live coding, live coding and the relevance of the live coding manifesto

A question that arises at almost every live coding conference or gathering: when is live coding still live coding when the “from scratch” concept is no longer a signature move, where does the line for what is still live coding get drawn? In our discussion, a key point which arose was the out of date nature of the TOPLAP Manifesto which is still often the primary reference when starting this discussion. Whilst the manifesto is a crucial historic document for live coding, when it was conceived there was a lack of critical vocabulary to describe live coding as the art form had just emerged. Perhaps it is time to sit down and rethink the manifesto to reflect where live coding is today, where it has come from and where it will go so that we are not still mindlessly quoting “show us your screens”! After all, what are we proposing with live coding? What affordances can live coding offer to the artist that a different artistic practice may not allow?

2.2 the music and language trends in live coding

Live coding has drawn in a wide range of people from both musical, visual and computer science backgrounds. However, just as easily as it has drawn some of us in, it has also pushed some of us out. There was a feeling in this cantina that Algoraves tend to dominate the public image of live coding and that more live coding events that present non-dance style music would benefit from getting the same spotlight. One of the issues that was brought up is that Algoraves presents a more homogenous use of technologies within live coding and do not represent the broader live coding communities. Furthermore, the English language is still at the root of the majority of live coding environments. If more non-english based live coding platforms would be developed/promoted, perhaps the community could be more inclusive. In the future we also wonder, could there be a google translate type service for different live coding languages?

2.3 the performance practice of live coding

Considering the majority of live coding performances are done via a computer, the issue of stage presence is one we found of importance. Perhaps a conventional stage-audience setup is not best suited for a live coding performance? Should live coding performances always be considered as an audio-visual experience or is this dependent on the setting and context. At times, not viewing the code can also heighten the audience's experience whilst at others, how are we to experience the “live” in live coding and, now to shamelessly quote the manifesto, “have access to the performer’s mind”? And where does “show us your screens” leave visualist live coders who’s code and visuals make use of the same visual space? Perhaps placing more importance on the coding editor used could enhance the performative experience. If the editors better reflected the gestural elements of the coding or became agents in themselves by placing limitations on how you code so as to remain legible, this could contribute to the performance practise. Tangible controllers, physical instruments and hybrid settings offer another approach to live coding performance presence.

2.4 Other topics

Visuals and coding is another theme we felt is under discussed in the current live coding discourse. Visuals are often used to enhance the music and rarely take centre stage - can we explore other relationships between visuals and music? Finally, the majority of visuals produced within live coding is abstract, in the cantina there was interest in creating non abstract movie style live coded visuals. Other topics brought up in this session included reflections on the current means of communication within the live coding community - primarily the shift from a centralised TOPLAP forum to language specific discord channels. If one assumes transparency as a live coding principle, does “faking” the code during a performance due to technical failure remain true to the art form? Will technologies evolve so much in the coming years that the conversations we are having now no longer make sense? Is sharing code on Github (or similar) post performance really reflective of the performance in any way or what are other means of documenting or saving performances that have taken place? Finally, in the spirit of a cantina, we would like to have more live coding recipes and cooking events.

How alive is a live-streamed Algorave

While even before the Covid-19 pandemic various occasions for live streamed live coding performances existed - e.g. live streams from individuals, live streams to on-site venues in order to reduce travel costs or circumvent visa restrictions or live stream events such as the TOPLAP birthday streams - in pandemic times live streams widely became the only possibility to do live coding performances for a public audience. More than one year after most public events in Europe were shut down all participants of this cantina session experienced live streamed live coding performances in some way, most of them both as audience as well as artists.

3.1 Cheating

The group discussed how streaming a pre-recorded live coding performance could be considered cheating. This question is generally valid for any kind of performance that is labelled a live stream, but it might be a more relevant question for live coding performances, considering live is already a part of live coding. Errors, glitches and slip-ups are usually embraced and considered to be integral parts of live coding performances. When there is the opportunity for the performance to be recorded multiple times until it is “clean” enough this error culture is missing. The discussion went further, even discussing whether pre-produced audio samples or loops should be considered cheating as well. It would be interesting to investigate if a notion of “cheating” in the context of live coding performances does implicate certain rules or at least expectations within the community. As a side note a benefit of live streaming came up: A recording of the performance is often created and published automatically alongside the stream, creating a vast archive of live coded performances.

3.2 Live streams as community activity in pandemic times

Notable were reports from the TOPLAP Barcelona community about their series “La hora del LIVECODER” (and La hora del live coder II) where every day a member of the community did a livestream at 8:08 pm. Participants reported how this did not only enable the local community to maintain their artistic practice but it reportedly also served as a highlight of the day - something to look forward to - for some participants in these emotionally difficult times of the first lockdowns, maybe even more contributing to the feeling of belonging to a community than with regular events occurring in a physical space.

3.3 The virtual spaces of a live coding live stream

Instead of the concert stage or a nightclub the live streaming live coder and its audience populates virtual spaces. Noted examples are the VR Algoraves where the environments were exclusively designed for the occasion and both artists as well as audience can express themselves through their choice of avatars. But also streaming websites like Twitch or YouTube create their own space, especially with its chat rooms and comment sections which create new ways of audience and audience/artist interaction. Via chat and comments, in contrast to physical events, it is possible to talk to every attendant at the same time. This also extends to hybrid events where the physically-present audience can still use the new communication channels using their phones, addressing everybody instead of only being able to talk to a person right next to them. Visual aspects of the projected user interface of a live coder can already be an important aspect of a live coding performance. A live coding live stream does often also incorporate a webcam - often portraying the face of the performer. A question arose if the face was actually the most relevant part of the body to show or if one should not rather display the hands. Likewise the sound of typing was mentioned as an - depending on the size of the space - often relevant acoustic property of a live coding performance that is usually not included in a live stream Except when made explicit, e.g. https://marijebaalman.eu/projects/code-livecode-live.html.

3.4 The physical spaces of a live coding live stream

Live coders usually perform for the space that they are currently in. When performing in a live stream this can still be partially true, e.g. a live stream from a living room observed in another living room. It came up that the style of music played in the live streams changed over time from the music that was performed in physical spaces. Especially artists that usually would perform dance music at Algoraves turn more into doing ambient/calmer music. There were also reports of people meeting in private living rooms to watch a live stream and have some drinks together. In that regard some social aspects of a live event were brought back, including opportunities to introduce friends to live coding by taking them along to an event. These gatherings in order to watch a live coding event together are not unlike watching a sports event, so jokingly (Or not?) the group imagined how it would be to have a commentator on a live coding stream, comparable to a soccer commentator. Live streaming from non-conventional sites and how it could affect the performance was also mentioned. One example given was Lucy Cheesman’s stream from next to a stream for the TidalCycles Winter Solstice marathon of 2020.

3.5 Issues with live streaming live coding performances

Also traditional issues of networked performances were discussed. In most settings latency did not matter much as the audience feedback via chat still works well even if it is slightly asynchronous. There was a general agreement that the degradation of the audio signal due to lossy data compression is not a big issue for experiencing the live coding performance. While text interfaces still look good even at lower frame rates and at lower resolution live coders doing visuals noted that many styles of visuals suffer greatly from video compression. On top of that, depending on available hardware, live streaming can take up additional resources on the computer which can create performance bottlenecks. Live streaming adds an extra layer of complexity and issues with technology and connectivity arise regularly. For these reasons some artists experienced live streaming as more stressful, but overall the live coding community is very open and welcoming and failure is not an issue, sometimes even embraced and celebrated.

Live Coding Machine Learning

Machine learning is being explored within live coding. While most early live coding performances used simple sound generators and processors, such as sine waves, filters, etc., today it is increasingly common to hear performances, even those starting from scratch, using machine learning to perform specific tasks. For instance, creating clusters in a music database so that the performer can navigate an ordered space. The use of machine learning algorithms has implications in live coding practice. For example, real time training v.s offline training. Should machine learning algorithms have to be visualized? Are new instruments and practices emerging from the use of machine learning? Should we collect data on-the-fly? How we manage the trade off between big and small data. These questions represent different decisions that impose different restrictions and imply specific aesthetic consequences.

4.1 Think about

When using machine learning within live coding, the accessibility problem related with the dependence on expensive hardware appears. In contrast we could focus on small data. For example: "you can learn a lot from three points". Machine learning is about automation, but in live coding there is no such thing as "final optimisation". Is it possible to build Machine forgetting algorithms? Neural nets, for example, are really bad at forgetting things (they need to be retrained all over again). Machine learning automates processes, then what's the role of the human in the practice? Machine learning as a collaborator (maybe pretrained, or using reinforce learning on the stage). Even if you automate something you still need a skilled person to operate it. Even state of the art algorithms have a percentage of failure, so they need supervision Thinkin about creativity vs automations. Machine learning is about automation but also about "finding new possibilities". Be aware of the “Ironies of automation” where you focus on a single part of the process and stop seeing the whole. After all, what problem do we want to solve? What do we want to optimize? Do we think we can use AI to create new music? Or do we want machine learning algorithms as another "college" to collaborate with?

Modern technology should allow for "getting rid of the repetitive stuff to make repetitive music".

4.2 Real time v. s offline training

Having real time feedback of the learning algorithm during the performance is limited by the computer's processing capacity and the size of the datasets. Simplifying this to a one dimensional problem, on one hand we have already trained models over huge and carefully curated datasets which will probably produce outputs close to those expected. However, these models can not be trained in mid performance. On the other hand we have approaches using small data and centered in interacting with the algorithm in real time, nonetheless the dataset size has to be manageable by the processor and the number of parameters have to be cognitively. These reasons are pushing the community to write dedicated algorithms. However, the performer needs to embrace unpleasant surprises mid-performance as the results sometimes may not be as "accurate" as those resulting from offline trained models.

4.3 Visualising algorithms/on-the-fly data collection

When having a small dataset, the training can be included as a part of the performance and algorithms can be intervened and visualized. This, viewed through the lenses of the Manifesto draft, might be a way to provide insight to algorithms. However, as mentioned above, on-the-fly training (for example trying different parameter combinations of the algorithm to play with models that offer either contrasting or slightly different results) conditions both algorithm complexity and dataset size, but might add performance transparency. Another possibility is to play or collaborate with agents. For example Cibo (Jeremy Stewart and Shawn Lawson), CYFO (Shelly Knotts) and Cacharpo (Luis Navarro), which are respectively: an autonomous TidalCycles performer and collaborative agents that allow Shelly to not repeat herself as the agent suggest the less likely code to write and Luis to co-perform cumbia sonidera. In these cases, what we appreciate during the performance is the algorithm result and how the trained model behaves or interacts with the performer to achieve a specific task. What is presented to the audience is the "taste" of the algorithm. On the other extreme are performances that produce and feed the training examples on-the-fly integrating these processes as parts of the performance. This of course allows, up to a certain point, having algorithm transparency but might be strongly conditioned by the time that the data acquisition and curation needs.