Autonomous Live Coding: Summer of Haskell Project

This summer, I was fortunate enough to get the chance to work on a project as part of the Summer of Haskell. This allowed me to work on the TidalCycles (Tidal) live coding software: free/open source software for pattern generation with code. Often this is used to create musical pattern, although increasingly it has been used for other things too, e.g. pattern visualisation, weaving. It is a language that is regularly used in live coding, a subset of artistic programming which looks at live improvisation and performance of algorithms to produce artistic outputs such as music, visuals or dance (to name a few).

I was mentored for this project by Alex McLean, the creator of TidalCycles and was provided with some real insight into the innards of the software. I’ve also been making and performing music with TidalCycles myself for a little while now (since around 2017) and so I’ve come to know it well enough, and have in the past looked at ways to shape it around my own practice. As TidalCycles is a domain-specific language embedded in the functional language of Haskell, it makes writing functions on the fly a possibility for people wanting to customise their own pattern creation, if they have the time and interest to get into the Haskell side.

In this project, I wanted to both expand TidalCycles in new directions and tie it in with the work that I am currently doing on my own PhD based in computational creativity. My initial project proposal aimed to tie these together, whilst trying to learn some Haskell too by perhaps leveraging existing Haskell libraries.

Why create an autonomous live code agent? Why use Tidal?

Machine Learning models tend to deal with processes that are static. Typical symbolic representations are in MIDI format (which in itself is quite a limiting representation of musical content, usually reducing the complexities of music to only pitch number and velocity and therefore losing a lot of nuance around timbre and expression). Moreover, most are trained on scores of musical information: pieces in which the outcome has already been predetermined (usually by a composer). Machine Learning of Indeterminate Music is also an area that has been underexplored, perhaps due to the complexities that arise, however Tidal could be a particularly useful software to explore these ideas.

How do we want autonomous agents to behave?

One of the first things we considered was the behaviour of our autonomous coder- how do we want it to behave and, maybe more importantly, how do we want to interact with it? Is it going to be standalone "intelligent" agent (musical intelligence would, arguably, have to include a strong focus on emotional intelligence) or is it augmenting the work of the live coder? How much autonomy can we (and do we want to) give it? Something that was also discussed was whether the agent would behave constructively or destructively? Arguably, both are equally valid forms of creative input and the latter presents an interesting challenge too: most computationally creative systems tend only to focus on the creation of new material. But, depending on our definition of human creativity, can we posit destruction as a form of creativity too?

About the project

The initial project proposal submitted in April 2020 looked at creating functional machine learning algorithms in the Haskell language for autonomous generation of new, syntactically correct Tidal Code. Hasktorch is an ongoing project that aims to leverage the core C++ libraries shared by PyTorch for ML/neural network architectures written using Haskell. Although this might be useful in future iterations of the project, the work ended up moving away from this, partly due to the challenge of having to implement a new NN architecture for this specific project, which seemed a little out of scope for a three month project.

To initiate this project, my mentor Alex also put me in touch with members of the community who were already working on similar projects during the community bonding period. This allowed me to reach out to others who were working in the area already, as there have been a few projects that have attempted to tackle this problem. I also started the Summer of Code by seeing what other things had been going on in the area by posting a thread on the TidalCycles forum. This flagged a few various interesting things, including an autonomous agent in Python using sequence-to-sequence neural network algorithm to autonomously generate Tidal Code and grammar-based agents, such as this auto-evaluating autonomous coding agent or evolutionary-based algorithms for autonomous code generation.

Tidal Development

Getting involved in the open source development of TidalCycles was one of the first things that I did during the summer of code, allowing me to understand more about both the contribution process and the innards of how TidalCycles applies the Haskell type system to represent the generation of patterns. I worked on fixing a couple of issues, like altering and adding new functions for better representations of an existing function, working on running tests for functions and changing the input type for other functions.

Further to this, I started working with my mentor on some work on development of a Tidal API. This was an experimental project that was creating a Haskell-based listener and receiver for interacting with Tidal. This followed the need to support messages sent between Tidal and autonomous agents running in a different process and/or language, but turned out to also be useful for human user interaction, allowing more feedback between user and software. For example, the user could run code and error check (where > represents an outgoing message and < an incoming message):

> /code <id> <source>

< /code/ok

expand the notation into a different format (e.g. from Tidal’s “mininotation” to Haskell):

> /expand <code> 

< /expand/ok <expanded code>

or get the cycles per second value

> /cps

< /cps <number>


As the work previously mentioned by Stewart et al. seemed particularly close to what I was initially proposing for the Summer of Code, we got in contact with them to arrange a few meetings to discuss any areas of overlap and potential collaborations. Their work however looked at building the architecture within Python, leveraging the tensorflow library and neural nets used commonly for speech translation problems. Rather than simply attempt the problem again within Haskell, we decided to alter some of this brief, perhaps looking to see how we could leverage some of their work to build a creativity support tool for Tidal users looking for some creative inspiration. The idea that came from discussions was building an "autocomplete" agent, one that could take an incomplete string of TidalCycles code and return suggestions of functions or parameters

Leveraging the existing work done by Stewart and Lawson meant adapting some of the code for use in an autocomplete agent became part of the scope of this work. To do so, in collaboration with the other researchers, we ended up building a repository for the two-way communication between the machine learning agent in python and the atom editor commonly used for evaluating TidalCode. This was done building a node server in atom and a mock python server, and hot key evaluation to select certain lines/words from the editor and send to the python server.

As the auto-completer’s ML backend was running in Python and we were sending OSC messages between a test python server and currently only an atom editor, the earlier discussed Tidal API would attempt to make this interaction more transparent and universal.

Tidal Walker

One issue flagged with the autocomplete agent was the nature of the corpus used for generation of new pattern, based on the code of a small set of live-coders. This would be able to create ‘statistically probable’ code, which might not have been seen before but would be unlikely to introduce radically new ideas. We wanted a way to expand the ‘search space’ of the machine learning agent, by adding combinations of functions to the corpus that had not been seen before.

This led to a related work that used random walks, Haskell’s strict type-checking system and n-gram models from a corpus of existing patterns, to create a potentially infinite number of possible Tidal patterns. The ‘walker’ takes a haskell type signature as a target – in this case a pattern of synthesiser control messages, and chooses a function that can ultimately produce that target. It then recursively fills in any missing arguments of that function. This takes account of partial evaluation, and the specialisation of polymorphic types as arguments as they are applied.

The aim was to create a walker agent that could navigate through the conceptual space of all possible Tidal code, modelling Haskell’s type system to ensure anything generated would be syntactically valid. To ‘steer’ this walk, we weighted its decisions based on n-gram data, created realistic code that could then be used to train the autocomplete agent. There were various steps taken to implement this. Below shows the architecture of such a program.

Tokenising Code

One of the first things that had to be done to achieve this was to correctly tokenise the patterns for the ngrams to be generated upon. The tokenisation process is described as below, with various functions having been written in Haskell to achieve the various steps.

Calculating Ngrams on Data

Once the dataset was correctly tokenised, it ended up being used to create the weightings for the ngram walker as follows. The tokenised code is then imported into a separate Haskell module, the frequency of ngrams is generated by calculating the frequency of occurrence of n-pairs of functions in the tokenised data. From the frequency distribution of n-pairs of consecutive functions, this can then sort these into a data structure that can be accessed later on by the walker to calculate the associated transition probabilities.

Weighted Walking for Unique Outputs.

Once the code has been tokenised and analysed, it can be used by a walking agent to generate new sequences of syntactically correct code. The inherent type signature structures are used as a framework for generating all the possible options that can occur after an initial function is chosen at random, thus ensuring the final code output generated can be evaluated. The weightings are then applied to their corresponding potential next functions in the sequence given the current function is known from the ngram data structure (this is applied as a Maybe value to denote that values are optional as there may not be some corresponding values from the dataset for the set of all possible next functions).

A variable is also implemented that could be a user controlled feature of any further iterations of the project, called "adherence." This allows control over how the weightings are structured, whether they adhere to the corresponding dataset, especially wherein a "Nothing" data type is generated for functions that are possible but unlikely.

Future Works

Currently, the tokenisation is only working on the Haskell functions of the TidalCycles domain specific language. There is, however, another aspect to writing TidalCycles known as its mini-notation. In brief, the mininotation is a mini-language within Tidal, allowing polyrhythmic sequences to be efficiently expressed. They are denoted by overridden strings, which are parsed into Tidal patterns. At this stage of this project, these strings have been treated as single tokens. However, this does not capture the inherent structures from the mini-notation and future work should look at how to correctly tokenise and model these too. Perhaps one way around this would be in creating a function that could expand these mininotation expressions directly into executable Haskell code.

Furthermore, the ngram dataset at the moment is currently only generating bigrams (ngram where n=2) giving limited higher-order representation. E.g. the output of a bigrams might show the transition probabilities for functions as:

slow 8

whereas an ngram of higher order (n >2) gives a wider representation of what is happening in the pattern:

range 0.1 0.2 $ slow 8 $ sine

Similarly, other functions e.g. "weave" takes a high number of functions for achieving syntactical correctness which a bigram couldn’t directly capture. Furthermore, ngrams do not capture the hierarchical structure of a syntax tree – looking into applying hierarchical Markov models to the abstract syntax trees could be a better approach, and is also left for future work. Nonetheless, bigrams, in capturing the likelihood of two tokens appearing together, does appear to improve the overall coherence of the generated code.

Some of the issues encountered with the output of the walker’s generation were similar to problems encountered in genetic programming. One of these is the notion of bloat (i.e. where various functions are added together that end up cancelling each other out like a series of successive (+1) and (-1) functions). In the Tidal generation this occurs where a pattern could be reversed 2n or 2n+1 times and this would be the same as an identity or the reversed pattern. Another issue encountered was idempotent functions, (i.e. where f(f(x) = f(x)). This occurs in cases such as every 1 (f(x)), which means that the function would be applied as without the every. Creating a filter for these would increase the efficiency of outputted code.

Project Reflections

Other than achieving outcomes from the project, some of the main things I have learnt have been to do with collaborating on an open source project. Not coming from a traditional computer science background, this has been one of the first larger scale projects I’ve been able to work on (although what I do has meant I’ve learnt a lot of computer science over a short period of time, I still feel “new” to the area). Some things that have been particularly useful to learn are using github to collaborate on coding projects (and how to fix merge conflicts!) and on attempting to meet milestones throughout the project, including trying to organise these coherently. 

One particular problem I encountered throughout was that most of my coding experience comes from imperative coding. This meant that when trying to write code in a functional way seemed a little unintuitive for me. It took some time to work on using concepts like folds and recursions rather than concepts like iterating over a for-loop. Learning Haskell through completing a project has definitely helped me think in a more functional way.

Finally of course, there were some errors that were encountered with the project. Debugging in Haskell was a little tricky without being able to backtrace/ find breakpoints in the code. Using a lot of print statements helped conceptualise how the program was running and where things might have been going wrong.

Tidal club course

It’s not all about actual tides..

I’ve started a “Learning TidalCycles” course with support of Lucy Cheesman (aka Heavy Lifting) and other friends (newcomers – see below for video explaining what live coding and TidalCycles is). We called it “Tidal Club” after some peer learning events Lucy started in Sheffield. The course is now in the fourth week of the first group of lessons. In an effort to keep the course accessible as possible, there’s absolutely no time limit on getting through the material, which is oriented around pre-recorded video and worksheets. (more thoughts about making online courses accessible here..)

So registrations are still open, and you can join the course it at any time, (although I might have to pause signups from time to time so we don’t get overrun with new learners). Interaction is based around the Tidal Club forum, for which we’ve got a code of conduct, and Lucy has been developing guidelines for giving each other constructive feedback on our work too.

The course is “pay as you feel” via this shop, with the following guide for the first four-week block:

  • £0 – for those who wouldn’t be able to participate otherwise
  • £12 (£3 per week) – standard
  • £24 (£6 per week) – those with extra cash to spare
  • £40+ (£10 per week) – those with institutional backing

Here’s the video lessons covered so far, at the time of writing (6th May 2020):

  • Intro
    • Installation
    • Technical tour of a Tidal system
  • Week 1 – mini-notation week!
    • Tidal Interaction
    • Loading sample packs
    • Sequencing with mini-notation (parts one, two and three)
  • Week 2
    • Starting out with effects
    • Manipulating time with setcps, cps patterns and fast/slow functions
    • Combining patterns with arithmetic (plus the ‘hurry’ function)
  • Week 3
    • Exploring the ‘every’ function, including tackling that ‘$’
    • ‘cut’ vs ‘legato’
    • ‘slice’ and ‘splice’
    • ‘chop’ and ‘striate’
  • Week 4 (this week)
    • Continuous ‘waveform’ patterns – sine, square, tri, saw and random functions (with a bit on binary patterns)
    • More still to come – all the random functions, including shuffle, sometimes and someCycles
  • Plus!
    • Challenges! (so far, exploring big tempo changes, and making music from speech)
    • Live streams (at different times to reach different timezones, and archived for later viewing)
    • A performance talk-through with Antonio Roberts
    • + more to come, including reference material.

After this four-week block we’ll take a rest, so I can prepare for the next four-week batch of lessons, taking things to the next level.. For which there’ll be another opportunity to pay-as-you-feel!

If you’re unsure what live coding and/or Tidal actually is, then here’s a quick video intro about that (and the also awesome Gibber and Hydra live coding environments), which I did for the creative guild in Sheffield:

If this sounds good to you, sign up over here!

(If you don’t want to join the course, but want to join the Tidal Club forum, you can register here.)

Tidal futures

Work has started on a bit of a rewrite/refactor of Tidal, I thought I’d write a quick post about how things are progressing. There’s a github project here if you’d like more info or want to get involved.

Structure comes from where you want

In current Tidal, “the structure comes from the left” has become a bit of a mantra. In the next version, it’ll come from where you want.

For example `”1 2 3″ + “4 5″` used to give you "4 6 7", because the structure comes from the first part, and  that2 in the middle starts in the first half of the cycle, so matches with 4.

In the rewrite, by default, structure comes from both sides, so in the above example, you end up with the equivalent of `”1 [6 7] 8″` — the `2` gets split in two, to match with both `4` and `5`.

To get the old behaviour, of structure coming from the left, you’d use |+, i.e. "1 2 3" |+ "4 5". To get structure from the right, you’d do "1 2 3" +| "4 5", which would give you "5 7".. Then `|+|` does exactly the same as +, i.e. it gets structure from both sides.

Unified pattern operators

|+|, `|+` and +| all work on patterns of parameters (now known as control patterns) as well as patterns of numbers. So these two are the same: `n (“1 2” + “3 4 5”) + s “bass”` and n "1 2" + n "3 4 5" + s "bass".

Of course there are also *, /, `%` and `-` versions of the above.

There is also `|>|` and `|<|` and friends; the arrow there points at where the values come from. So you could do `”[1 2] [3 5 6]” |> “5 2″` to take the structure from the left, and the values from the right, giving you `”[5 5] [2 2 2″`. Aha!

|> basically does what `#` does now, we’ll likely keep that as an alias..

On the way.. Stateful patterns

Patterns can now respond to state, the idea being that you can do things like `note “c a f e” # sound “bass” # speed (scale 1 1.6 $ control “knob1”)` where knob1 somehow maps to an external controller. Latency might be an issue.. It’s all in bits at the moment though, the new version doesn’t actually make sound yet.. Will update this post as things develop.


The most exciting thing for me is that the code is a bit cleaner on the inside, with fewer bugs and edge cases, and things generally make more sense. There’s even some tests, a start at creating a more complete definition of what patterns are and how they should behave. Patterns are flagged as being either continuous or discrete, which clears up a lot of ambiguity that was around before.

Keep an eye on the project for updates!

TidalCycles Extension 0.5.6 for VS Code

The TidalCycles extension for VS Code has a fresh update. A couple of improvements have been made around configuration setting errors and custom boot file path errors.

Config Setting Errors

The new update now removes errors telling you that the TidalCycles config settings were “unknown”. Now it’s just a clean config:


Boot File Path Errors

A few minor versions ago, we introduced the ability to configure a path to a custom Tidal boot file. With that, you can now configure Tidal to load up all of your custom stuff in a seamless bootup sequence. However, if VS Code could not locate your custom bootup file it wouldn’t tell you.

Now, the Tidal console output and a VS Code message will notify you that you boot file was not found:

What’s more is that you can fix your boot file config settings and try to reboot Tidal without restarting VS Code. Fancy!

Oh Hey One More Thing!

VS Code recently released a new feature where you can dock the terminal to the right-hand side of the IDE:

So now you can put the Tidal output to the side instead of below your Tidal code, if you so desire:

TidalCycles 0.9.8

TidalCycles 0.9.8 newly out! It replaces 0.9.7 which didn’t work well with many haskell versions. Here’s a quick run through the changes since 0.9.6. Any problems or questions, join us on the #tidal channel on, or the tidal list on

How to update

If you use cabal (rather than stack) to install, it’s just a matter of running cabal update then `cabal install tidal` from the commandline. I’m not clear on the best way to update using stack (as tidal hasn’t yet updated on stackage LTS), if you know please leave a comment!

New Bjorklund (aka Euclidean) functions

New function einv, which fills the “blanks” left by e. For example, e 3 8 "x" gives "x ~ ~ x ~ ~ x ~", whereas einv 3 8 "x" gives `”~ x x ~ x x ~ x”`.

Another new function `efull` which takes two patterns rather than one, combining `e n k` on the first with `einv n k` on the second. For example, `efull 3 8 “2” “1”` gives `”2 1 1 2 1 1 2 1″`

Yet another function distrib, which is similar to e, but takes a list of numbers. `distrib [5,8] “x”` is the same as e 5 8 "x", but `distrib [2,5,8]` will  do something rather freaky.. It will take the `(5,8)` pattern as a starting point (which is "x ~ x x ~ x x ~") and then attempt to distribute two events evenly over those five xs as though they were contiguous,  creating `”x ~ ~ x ~ ~ ~ ~”`). Basically, give it a list of numbers which increase in value and you’ll find some interesting off-kilter rhythms.

`d1 $ distrib [5,7,16] $ sound “bd:7″`

Sequence parser

Now you can give * and / subpatterns in the parser. For example `”[a b]*[2 3]”` would be the same as "[a b] [b a b]" (i.e. the first half of "a b a b", which is the pattern at twice the speed, and the second half of "a b a b b a b", which is the pattern at three times the speed).

Floating point notes

Now notes are floating point. This means that you can do things like d1 $ s "drum*8" # n (sine * 8). It also means that for some synthesised sounds you can play between the notes (microtones?), e.g. `d1 $ sound “supermandolin*8” # n sine`. There have been other adjustments in this area, so previously where you had to do fiddly conversions between integers and floats, you no longer have to.

ghc support

Tidal now supports the latest version of ghc (8.4.1). You should probably be running at least 7.10.3 by now but older versions may still work.


Exciting SuperDirt update to follow soonish, including new MIDI support.



TidalBot is back! You can tweet tidal patterns to @tidalbot on twitter, and it will give you back an mp3 of the pattern, as well as a PDF with a visualisation of it.. Here’s a couple of examples:

As a bonus, the latest pattern is currently being projected into the shop window of Access Space Labs on Fitzalan Square in Sheffield.

Atom Package v0.11.0: custom Tidal boot path

A new release (version 0.11.0) of the TidalCycles package for Atom is available today. With this version you can now specify a path to your own boot file in the package settings:

This means that you no longer need to modify the provided BootTidal.hs file in the package with your own custom logic (e.g. custom functions, additional modules, etc).

If you want to create your own boot file:

  1. Create a copy of the original (available here)
  2. Save it anywhere on your computer
  3. Modify the TidalCycles package settings to point to where you saved the new file

How to Haskell – developing a Markov chain

One of my favorite things about Tidal is that it’s inside of Haskell, a well-developed full-blown programming language, so all sorts of tricks are available.  After some discussion on Slack I developed code for making Markov chains to use in Tidal, and here I’ll try to describe the process.

First – what’s a Markov chain? Basically it’s some kind of random sequence where each step depends only on the step immediately before it. A common way to express this is to imagine several possible “states” for each step, and then write a “transition matrix” which gives the odds of going from one state to another.

How should we represent all this in Tidal/Haskell?  We ultimately want to have the chain as a Pattern, but it’ll probably be easier to start out with an ordinary list, and then use the listToPat function to turn it into a Pattern.  Let’s number the states with integers, starting at zero.  Then the chain itself will be a list of integers.  The transition matrix is a list of lists of floating point numbers – we’ll use doubles. Then one way of writing things is to have a function that takes the transition matrix and a chain as arguments, and returns a new chain which is just the old one with one new element added. Here’s just the signature:

markovStep :: [[Double]] -> [Int] -> [Int]

Next let’s start the basic structure of what the function will have to do. Let’s call the transition matrix tp and the input chain xs. We need to look at the top element of the list to see what the “from” state is, and find that row of the transition matrix: tp!!(head xs). Next we need to generate a random number r (we’ll worry about how to actually do this later) and somehow use it to choose what the “to” state for the next step of the chain should be.

For example, if the row of the matrix were [0.20, 0.30, 0.50], this would mean 20% chance of transition to state zero, 30% chance to state one, and 50% chance to state two. So if our random number (between zero and one) were between 0 and 0.2, we’d pick state zero, between 0.2 and 0.5 state one, etc. This is easier to write if we take the cumulative sum of the row, and there’s a handy Haskell function we can use for that: scanl1 (+). That’ll turn the row into [0.20, 0.50, 1.00], and now when we have a random number we just need to find the index of the first element in the row larger than our random number. That’ll be the state the chain jumps to next.

There’s a handy Haskell function for finding things in lists too: findIndex. While it doesn’t quite work yet, we have most of our function:

markovStep tp xs = (findIndex (r <=) $ scanl1 (+) (tp!!(head xs))) : xs

(the : xs at the end adds the existing chain after the new element we just made)

One problem is that findIndex doesn’t actually return an Int. It returns a Maybe Int, because it might not find anything.  If our transition probabilities for each row add to one we’ll always be safe, but if we mess up or make a typo things might go wrong.  We have two options, (1) choose a safe default or (2) throw an exception and halt.  For the first option we could use fromMaybe 0 (to default to state zero), and for the second we’d use fromJust. Both require us to import the Data.Maybe module. Since I like to immediately know when I’ve made a math mistake and don’t mind if my performance grinds to a halt, I’m choosing the second option:

import Data.Maybe

markovStep tp xs = (fromJust $ findIndex (r <=) $ scanl1 (+) (tp!!(head xs))) : xs

The final thing to take care of is the random number. While it’s not quite the intended use, we can use the timeToRand function, which takes a rational number as an argument and hashes it to a pseudorandom result. We just have to make sure we get a different number for each step of the chain by using the length of the chain, and do type conversion as necessary:

import Data.Maybe

markovStep tp xs = (fromJust $ findIndex (r <=) $ scanl1 (+) (tp!!(head xs))) : xs 
  where r = timeToRand $ fromIntegral $ length xs

The last touch to make this a bit more usable is to wrap markovStep in a function that will repeat the step process to build a chain n numbers long from an initial state xi, flip it around so the most recent element is at the end, and turn the whole thing into a pattern with one number per cycle.

import Data.Maybe

markovStep :: [[Double]] -> [Int] -> [Int]
markovStep tp xs = (fromJust $ findIndex (r <=) $ scanl1 (+) (tp!!(head xs))) : xs 
  where r = timeToRand $ fromIntegral $ length xs
markovPat n xi tp = slow (fromIntegral n) $ listToPat 
  $ reverse $ (iterate (markovStep tp) [xi]) !! (n-1)

Now it’s ready to use in Tidal:

 $ fast 4 
 $ n (markovPat 64 0 [[0.3, 0.4, 0.3], [0.3, 0, 0.7], [0.7, 0.3, 0]])
 # s "arpy"

The chain states are always integers starting at zero, and that seems rather limiting.  But we can treat the chain as an index to some other array of our choice, which needn’t even be numerical:

 $ fast 4
 $ n (fmap ([0,3,5]!!) $ markovPat 64 0 [[0.3, 0.4, 0.3], [0.3, 0, 0.7], [0.7, 0.3, 0]])
 # s "arpy"

 $ fast 8
 $ s (fmap (["hc:2", "cr"]!!) $ markovPat 128 1 [[0.88, 0.12], [0.7, 0.3]])
 # cut 1

I’m sure there are still improvements we could make, but this is a pretty fun start!

TidalCycles Winter Solstice, Sheffield UK

We’re having a Winter Solstice party in Sheffield on 21st December at Access Space, tickets now up! Featuring some of the leading lights from the local tidalcycles community Heavy Lifting, Tich, Digital Selves, Class-Compliant Audio Interfaces, Co(n)de Zero and Nullish. It will get algorave-y at some point.. There’s a facebook event for those who are that way inclined..