MahlerNet
It all started when I saw what the Deep Dream Generator could do... Never before had I seen a computer generate something that so deeply struck me as genuine and original art. Being a musician with a bachelor in composition, a seed was planted deep in me with the curious question of what neural networks could do with music. I was especially thrilled by the idea that a neural network could discover latent factors in music and combine them in the most marvellous ways so that we could hear what a mix between, for example, Queen and Mozart would sound like.
***
My name is Elias and I used to be a student at the Royal College of Technology in Stockholm, Sweden. When it was time for me to write my master's thesis, I started out by making a thorough literature survey in the field
of algorithmic composition with neural networks. After all, I did not know much about this field at that point. By the time I was done, by coincidence, one of the heroes from the literature, Bob Sturm, showed up as
an associate professor at my university and I convinced him to sign on as guidance on my sail towards higher knowledge. Bob has a place in the history of algorithmic composition with neural networks with his folk-rnn
which uses language model LSTMs to model music in ABC format. When it was time to design a new network, I combined my own experiences from music theory along with the best ideas I had encountered in the literature,
and implemented a network that I eventually decided to try on Mahler; my favourite composer of all time. Maybe I did not reach all the way to where we get to hear a mix of Queen and Mozart with this project, but
there are some light moments which I hope to share with you on this website, whose main purpose is to provide listening examples. Samples can be listened to as is, or while reading the thesis to find out more
about the experiments. Some of the samples are noisy so turn down the volume before you know if you like it or not. Samples marked with a 
are
samples that I am particularly fond of. For samples to work in Edge and Explorer, the proper media feature pack for Windows must be installed.
***
MahlerNet is a neural network that models symbolic (written) polyphonic music with multiple instruments in MIDI format that was conceived as part of a master's thesis at the Royal College of Technology in Stockholm, Sweden. MahlerNet uses a custom MIDI preprocessor in which several heuristics from music theory are implemented. Furthermore, MahlerNet builds on the previous architectures of MusicVAE, PerformanceRNN, BachProp and BALSTM and was implemented from scratch in Python and Tensorflow in the spring of 2019.
Hope you enjoy it!
Elias Lousseief
*Later on in 2019, me and Bob wrote an article for SMC2019 (1st Nordic Sound and Music Computing Conference 2019) on the most important findings of my master's thesis which was accepted! Yay!
**Later on in 2020, my work with MahlerNet earned me the SAIS (Swedish AI Society) Master's Thesis Award and a contribution to participate in an AI related event of my choice. The prize was shared with Ulme Wennberg whose thesis contributed to the popular field of Natural Language Processing. Congrats to us! Yay2!
The code to MahlerNet is available at the MahlerNet Github page: 
The thesis on MahlerNet can be downloaded here: 
The SMC2019 article on MahlerNet can be downloaded here:
The samples presented below can be downloaded here: 
My Curriculum Vitæ and portfolio:
Contact: music@mahlernet.se
***
Experiment 1 with pitch only
In the first experiment, only pitch is modelled and hence, a default of an eighth note or triplet eighth note is used as offset and duration while piano is used as instrument. In the interpolations, the first and last bars are the original bars that the interpolation is between, thus they may contain other offsets and durations. Interpolations are in general smooth and random samples are plausible. Most examples present from this experiment might be fairly uninteresting, but they are present as a complement to the thesis. Samples 16-21 are without conditioning, thus no structure is expected here, to the contrary of samples 22-27 where conditioning has been used.
Data set(s): ESSEN, SESSION
Sample 1-1
Type: 10-step interpolation
Trained on: SESSION
Sample 1-2
Type: 10-step interpolation
Trained on: SESSION
Sample 1-3
Type: 10-step interpolation
Trained on: SESSION
Sample 1-4
Type: 10-step interpolation
Trained on: SESSION
Sample 1-5
Type: 10-step interpolation
Trained on: SESSION
Sample 1-6
Type: 10-step interpolation
Trained on: SESSION
Sample 1-7
Type: 10-step interpolation
Trained on: ESSEN
Sample 1-8
Type: 10-step interpolation
Trained on: ESSEN
Sample 1-9
Type: 10-step interpolation
Trained on: ESSEN
Sample 1-10
Type: 1-bar random sample
Trained on: SESSION
Sample 1-11
Type: 1-bar random sample
Trained on: SESSION
Sample 1-12
Type: 1-bar random sample
Trained on: SESSION
Sample 1-13
Type: 1-bar random sample
Trained on: ESSEN
Sample 1-14
Type: 1-bar random sample
Trained on: ESSEN
Sample 1-15
Type: 1-bar random sample
Trained on: ESSEN
Sample 1-16
Type: 10-bar seeded sample
Trained on: SESSION
Sample 1-17
Type: 10-bar seeded sample
Trained on: SESSION
Sample 1-18
Type: 10-bar seeded sample
Trained on: SESSION
Sample 1-19
Type: 10-bar seeded sample
Trained on: ESSEN
Sample 1-20
Type: 10-bar seeded sample
Trained on: ESSEN
Sample 1-21
Type: 10-bar seeded sample
Trained on: ESSEN
Sample 1-22
Type: 10-bar seeded sample
Trained on: SESSION
Sample 1-23
Type: 10-bar seeded sample
Trained on: SESSION
Sample 1-24
Type: 10-bar seeded sample
Trained on: SESSION
Ends on tonic.
Sample 1-25
Type: 10-bar seeded sample
Trained on: ESSEN
Sample 1-26
Type: 10-bar seeded sample
Trained on: ESSEN
Sample 1-27
Type: 10-bar seeded sample
Trained on: ESSEN
Motive reiterated twice at the end.
***
Experiment 2 with all properties
In the second experiment, all the properties of offset, duration, pitch and instrument are modelled. The training set PIANOMIDI contains polyphonic piano music from several periods whereas MUSEDATA contains polyphonic both piano music and orchestral music, mostly from the baroque and classic era. In interpolations, the first and last bars are copied from the original.
Data set(s): PIANOMIDI, MUSEDATA
Sample 2-1
Type: 10-step interpolation
Trained on: PIANOMIDI
Good interpolation despite many events in end points.
Sample 2-2
Type: 5-step interpolation
Trained on: PIANOMIDI
Another decent interpolation with many events.
Sample 2-3
Type: 1-bar random sample
Trained on: PIANOMIDI
Musically implausible random sample.
Sample 2-4
Type: 10-bar random sample
Trained on: PIANOMIDI
Musically implausible random sample.
Sample 2-5
Type: 10-step interpolation
Trained on: PIANOMIDI
Sample 2-6
Type: 5-step interpolation
Trained on: PIANOMIDI
Interpolation with similar endpoints but in different registers. Notice how the register is gradually shifted.
Sample 2-7
Type: 1-bar random sample
Trained on: PIANOMIDI
Sample 2-8
Type: 10-step interpolation
Trained on: PIANOMIDI
Unsuccessful interpolation but rhythmically interesting with complemental voices.
Sample 2-9
Type: 5-step interpolation
Trained on: PIANOMIDI
Sample 2-10
Type: 1-bar random sample
Trained on: PIANOMIDI
Musically plausible random sample.
Sample 2-11
Type: 10-bar random sample
Trained on: PIANOMIDI
No conditioning so bars are independent and no structure is expected, but each bar is more musically plausible than several of the previous random samples.
Sample 2-12
Type: 10-step interpolation
Trained on: MUSEDATA
Unsuccessful interpolation where only rudimentary aspects of the endpoints are outlined.
Sample 2-13
Type: 1-bar random sample
Trained on: MUSEDATA
Another unplausible random sample ...
Sample 2-14
Type: 10-step interpolation
Trained on: MUSEDATA
Sample 2-15
Type: 1-bar random sample
Trained on: MUSEDATA
Sample 2-16
Type: 10-step interpolation
Trained on: MUSEDATA
Probably the most unsuccessful interpolation in all experiments. Hardly any aspects of the endpoints can be heard and notice that the second to last bar is not even remotely similar to the last.
Sample 2-17
Type: 1-bar random sample
Trained on: MUSEDATA
Sample 2-18
Type: 10-bar seeded sample
Trained on: PIANOMIDI
A different Für Elise ...
Sample 2-19
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-20
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-21
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-22
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-23
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-24
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Triplet structure preserved after seed via conditioning.
Sample 2-25
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-26
Type: 10-bar random sample
Trained on: PIANOMIDI
Sample 2-27
Type: 10-bar random sample
Trained on: PIANOMIDI
Eerie tune with preserved motives in the bass for quite a few bars.
Sample 2-28
Type: 10-bar random sample
Trained on: PIANOMIDI
Sample 2-29
Type: 10-bar random sample
Trained on: PIANOMIDI
Typical avant-garde 20th century piano music (with the exception of the brass fill).
Sample 2-30
Type: 10-bar random sample
Trained on: PIANOMIDI
Impressionist beginning followed by a tune with character.
Sample 2-31
Type: 100-bar random sample
Trained on: PIANOMIDI
Monotoneous at times but with contrasting parts and direction.
Sample 2-32
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Motivic figures inspired from the seed.
Sample 2-33
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-34
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Finding a groove towards the end.
Sample 2-35
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-36
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-37
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-38
Type: 10-bar seeded sample
Trained on: PIANOMIDI
Sample 2-39
Type: 10-bar random sample
Trained on: PIANOMIDI
Large dark chords resolved gradually and characteristically into a sweet tune.
Sample 2-40
Type: 10-bar random sample
Trained on: PIANOMIDI
Sample 2-41
Type: 10-bar random sample
Trained on: PIANOMIDI
Sample 2-42
Type: 10-bar random sample
Trained on: PIANOMIDI
Characteristic tune which seems slightly skewed because of subtle dissonances.
Sample 2-43
Type: 100-bar random sample
Trained on: PIANOMIDI
Despite a few implausible sections, contrast can be found as well as motivic passages. Notice the long terminating sequence that wanders in harmony but stays in style.
Sample 2-44
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-45
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-46
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-47
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-48
Type: 10-bar seeded sample
Trained on: MUSEDATA
Strings punctuated by timpani is sustained from seeds for several bars.
Sample 2-49
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-50
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-51
Type: 10-bar random sample
Trained on: MUSEDATA
Baroque tune with plausible contrapuntal and harmonic progression.
Sample 2-52
Type: 10-bar random sample
Trained on: MUSEDATA
Sample 2-53
Type: 10-bar random sample
Trained on: MUSEDATA
Some less successful parts but otherwise the same rhythmic flow lasts throughout.
Sample 2-54
Type: 10-bar random sample
Trained on: MUSEDATA
Baroque piano piece that starts out in a romantic way, ahead of its time. Very plausible complementary voices.
Sample 2-55
Type: 10-bar random sample
Trained on: MUSEDATA
Yet another baroque piano piece with plausible complementary right and left hand.
Sample 2-56
Type: 10-bar random sample
Trained on: MUSEDATA
Baroque piece with theme-like motives returning every now and then. Notice how the piece departs after the initial stationary bars.
Sample 2-57
Type: 10-bar random sample
Trained on: MUSEDATA
Classic piece starting out with high strings after which joins lower strings to build up, eventually with help from french horns. Clear cadenzas.
Sample 2-58
Type: 100-bar random sample
Trained on: MUSEDATA
Starts out as a plausible baroque composition with theme-like passages. Deviates after a while into unplausible but expressive music that still adheres to baroque style even though it is played at remarkable speed.
Sample 2-59
Type: 100-bar random sample
Trained on: MUSEDATA
Very plausible classic symphonic piece with contrasting sections in terms of instrumentation, dynamics and style. Notice well the short minor passage leading up to the homophonic section in the middle.
Sample 2-60
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-61
Type: 10-bar seeded sample
Trained on: MUSEDATA
Instrumentation and its development made coherent and plausible thanks to conditioning, despite poor overall musical plausibility.
Sample 2-62
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-63
Type: 10-bar seeded sample
Trained on: MUSEDATA
Sample 2-64
Type: 10-bar random sample
Trained on: MUSEDATA
Not a very classic harmonic development but yet interesting and somewhat plausible.
Sample 2-65
Type: 10-bar random sample
Trained on: MUSEDATA
Sample 2-66
Type: 10-bar random sample
Trained on: MUSEDATA
Sample 2-67
Type: 100-bar random sample
Trained on: MUSEDATA
Implausible baroque piece with contrapuntal aspects but nothing else in place.
***
Experiment 3 with all properties
In the third experiment, all the properties of offset, duration, pitch and instrument are modelled yet again. The training set used is now a custom training set with most of the movements of Mahler's symphonies. This music is orchestral and highly polyphonic but differs from baroque and classic music in that it contains more dissonances and more instruments. In interpolations, the first and last bars are copied from the original. Can you hear from which symphony and movement the seeds are from?
Data set(s): MAHLER
Sample 3-1
Type: 10-step interpolation
Smooth interpolation in a slow passage with few events.
Sample 3-2
Type: 10-step interpolation
Somewhat smooth interpolation, notice when the lower note of the rhythmic figure starts to change.
Sample 3-3
Type: 10-step interpolation
Somewhat smooth interpolation where it is easy to hear when the step is taken from the major chord in the start point to the minor of the end point.
Sample 3-4
Type: 10-step interpolation
Sample 3-5
Type: 10-step interpolation
Unsuccessful interpolation resulting in a typical romantic shimmering and emerging sound.
Sample 3-6
Type: 10-step interpolation
Unsucessful interpolation; notice how intermediate steps contain music dissimilar to both the music in the start and end endpoint.
Sample 3-7
Type: 10-step interpolation
Unsuccessful interpolation but the repeated brass chords bring about a dark and foreboding feeling.
Sample 3-8
Type: 10-step interpolation
Few notes in endpoint bars result in smooth interpolation.
Sample 3-9
Type: 10-step interpolation
Unsuccessful interpolation but interesting musically.
Sample 3-10
Type: 10-bar seeded sample
Sample 3-11
Type: 10-bar seeded sample
Brass chords and timpani continued after seed thanks to conditioning.
Sample 3-12
Type: 10-bar seeded sample
Smooth and sparse passage is prolonged from seed via conditioning
Sample 3-13
Type: 10-bar seeded sample
Sixteenth notes in woodwinds are continued after seed thanks to context conditioning.
Sample 3-14
Type: 10-bar seeded sample
Piece continues in similar style after seed thanks to conditioning.
Sample 3-15
Type: 10-bar seeded sample
Sample 3-16
Type: 10-bar seeded sample
Sample 3-17
Type: 10-bar seeded sample
Typical romantic horns wandering against a pad of violins towards the end.
Sample 3-18
Type: 10-bar seeded sample
Sample 3-19
Type: 10-bar seeded sample
Sample 3-20
Type: 10-bar seeded sample
Typical high Mahler strings setting the scene for sparse events in the foreground.
Sample 3-21
Type: 10-bar seeded sample
Sample 3-22
Type: 10-bar seeded sample
Sample 3-23
Type: 10-bar seeded sample
Sample 3-24
Type: 10-bar seeded sample
High strings and brass bring about a typical Mahlerish scene of relief and victory.
Sample 3-25
Type: 10-bar seeded sample
Low brass leading the way with help from the woodwinds.
Sample 3-26
Type: 10-bar seeded sample
Sample 3-27
Type: 10-bar seeded sample
Sample 3-28
Type: 10-bar seeded sample
Flow of eighth notes kept from seed via conditioning.
Sample 3-29
Type: 10-bar seeded sample
Triplets from seed survive many bars before resolved into chords.
Sample 3-30
Type: 10-bar seeded sample
Playful scene involving most instruments.
Sample 3-31
Type: 10-bar seeded sample
Sample 3-32
Type: 10-bar seeded sample
Sample 3-33
Type: 10-bar seeded sample
Sample 3-34
Type: 10-bar seeded sample
Sample 3-35
Type: 10-bar seeded sample
Sample 3-36
Type: 10-bar seeded sample
Sample 3-37
Type: 10-bar seeded sample
Sample 3-38
Type: 10-bar seeded sample
Sample 3-39
Type: 10-bar seeded sample
Plausible build-up to tutti with polyphony in woodwinds.
Sample 3-40
Type: 10-bar random sample
Sample 3-41
Type: 10-bar random sample
Sample 3-42
Type: 10-bar random sample
Vigorous, dramatic character swiftly changing into reminiscent nostalgy.
Sample 3-43
Type: 100-bar random sample
Starting with altering instrument groups but is drawn to stationary ambient homophonic setting.
Sample 3-44
Type: 100-bar random sample
A few different contrasting sections but with too long time spent on long homophonic passages with high strings.
Sample 3-45
Type: 10-bar seeded sample
Not very interesting musically but shows that context conditioning manages to create long term structure, here in tonality and instrumentation.
Sample 3-46
Type: 10-bar seeded sample
Sample 3-47
Type: 10-bar seeded sample
Sample 3-48
Type: 10-bar seeded sample
Sample 3-49
Type: 10-bar seeded sample
Instrumentation and setting remain after seed but not going anywhere harmonically.
Sample 3-50
Type: 10-bar seeded sample
Sample 3-51
Type: 10-bar seeded sample
Impressive and efficient rhythmic section emphasized by timpani towards the end.
Sample 3-52
Type: 10-bar seeded sample
Sample 3-53
Type: 10-bar seeded sample
Sample 3-54
Type: 10-bar seeded sample
Sample 3-55
Type: 10-bar seeded sample
Sample 3-56
Type: 10-bar seeded sample
Sample 3-57
Type: 10-bar seeded sample
Sample 3-58
Type: 10-bar seeded sample
Sample 3-59
Type: 10-bar seeded sample
Calm passage with sweet and warm horns painting the scene.
Sample 3-60
Type: 10-bar seeded sample
Sample 3-61
Type: 10-bar seeded sample
Sample 3-62
Type: 10-bar seeded sample
Sample 3-63
Type: 10-bar seeded sample
Repetitive motive emerging and sustained in the middle.
Sample 3-64
Type: 10-bar seeded sample
Instruments in unison towards the end very similar in style to the beginning of Mahler's first symphony.
Sample 3-65
Type: 10-bar seeded sample
Eighth notes and drive sustained from seed via conditioning.
Sample 3-66
Type: 10-bar seeded sample
Sample 3-67
Type: 10-bar seeded sample
Sample 3-68
Type: 10-bar seeded sample
Dramatic and engaged timpani towards the end.
Sample 3-69
Type: 10-bar seeded sample
Sample 3-70
Type: 10-bar seeded sample
Sample 3-71
Type: 10-bar random sample
Gentle and plausible development of a tender passage with the exception of some abrupts entrances.
Sample 3-72
Type: 10-bar random sample
Very dramatic resolution supported by timpani and sustained bass.
Sample 3-73
Type: 10-bar random sample
Sample 3-74
Type: 10-bar random sample
Sample 3-75
Type: 100-bar random sample
Unplausible sample that at least does not get stuck in homophonic sustained passages with no harmonic or motivic movement.
Sample 3-76
Type: 100-bar random sample
Long piece that loses credibility totally at occasions but also contains some nice moments.
***
Extras
The extras section contain a few pieces used to elaborate the discussion.
Sample 4-1
Type: 10-step interpolation
Trained on: MAHLER
Very smooth interpolation without deviation despite quite many events in endpoints.
Sample 4-2
Type: 10-step interpolation
Trained on: MAHLER
Another example of a smooth interpolation despite much content. Notice the long repeated rhythms in the end point that can be heard early on.
Sample 4-3
Type: 10-step interpolation
Trained on: PIANOMIDI
Worthless interpolation but quite nice musically. Success or failure?