Exploring Music Transcription with Multi-Modal Language Models | by Jon Flynn | Nov, 2024

Utilizing Qwen2-Audio to transcribe music into sheet music

Computerized music transcription is the method of changing audio recordsdata like MP3 and WAV into sheet music, guitar tablature, and any format a musician could need to study a track on their instrument.

We’ll go over the perfect present instruments for doing this, which occur to be deep learning-based, and a novel method for it.

The present state-of-the-art for this process comes from Magenta, an open-source analysis mission developed by the now defunct (as of April 2023) Google Mind Group.

They launched a paper Sequence-to-Sequence Piano Transcription with Transformers in 2021 which used a T5-inspired transformer mannequin (just like “t5-small”) with 54 million parameters and the Maestro dataset, reaching nice outcomes. The issue is approached as a sequence-to-sequence process utilizing an encoder-decoder Transformer structure. The encoder processes mel spectrogram frames as enter and produces embeddings, whereas the decoder makes use of these embeddings through cross-attention to autoregressively generate a sequence of MIDI-like tokens. Their vocabulary consisted of 4 sorts of tokens:

• Notice tokens (128 values for MIDI pitches)
• Velocity tokens (128 values together with zero for note-off)
• Time tokens (6,000 values in 10ms bins for absolute timing)
• EOS token (to mark sequence finish)

See the picture under for a visualisation of the structure and an instance sequence of their customized MIDI tokens:

Our mannequin is a generic encoder-decoder Transformer structure the place every enter place comprises a single spectrogram body and every output place comprises an occasion from our MIDI-like vocabulary. Outputs tokens are autoregressively sampled from the decoder, at every step taking the token with most chance.

In 2022, they launched a paper, MT3: Multi-Task Multitrack Music Transcription. This experiment used the identical method because the final one however added further instrument tokens to symbolize the totally different devices. Once more, they used an identical T5 mannequin and achieved nice efficiency in opposition to lots of the datasets educated on, notably Slakh, Maestro and MusicNet.

MR-MT3 was launched the next 12 months as a slight enchancment to MT3.

Compute/GPU sources

Enormous sources had been wanted to coach this from scratch, regardless of being a lot smaller in dimension in comparison with even the smallest language fashions. The 2021 paper famous:

“We educated all fashions on 32 TPUv3 cores, leading to a per-core batch dimension of 8. Based mostly on validation set outcomes, overfitting didn’t appear to be an issue, so we allowed coaching to progress for 400K steps, which took about 2.5 days for our baseline fashions.”

The MT3 paper doesn’t present as particular particulars on coaching, stating they practice for 1 million steps.

Different limitations

These fashions have some inherent limitations of their output flexibility. Whereas language fashions sometimes have giant vocabularies (typically 30,000+ tokens) which can be extensively pre-trained on various pure language information, MT3 and related music transcription fashions use a a lot smaller, specialised token vocabulary (only some thousand tokens) targeted solely on musical occasions. This specialisation implies that including new tokens, akin to for brand spanking new devices or taking part in methods like palm muting on guitars or pizzicato on violins, is probably going not simple — it requires vital retraining to combine these new tokens successfully with the prevailing vocabulary, and sometimes requires substantial coaching information demonstrating these methods. This differs from giant language fashions which might typically describe such musical nuances in pure language with out modification, as they’ve encountered these ideas throughout their broad pre-training.

Switch studying and zero-shot

We are able to leverage switch studying from giant open-source pre-trained audio and language fashions. Examples of music technology fashions embrace OpenAI’s Jukebox and Meta’s MusicGen.

GPT-4o is designed to deal with textual content, audio and pictures “natively”. Though OpenAI has not launched the technical particulars on this, it’s assumed that some weights within the community will course of all modalities. It’s potential that the mannequin makes use of a decoder-only structure like language solely GPT fashions with out the necessity for encoder parts to transform totally different modalities to a dense illustration first. This design permits the mannequin to seamlessly course of and interpret inputs like textual content and pictures collectively, probably providing efficiency advantages each computationally and when it comes to mannequin understanding.

Many multi-modal fashions take a less complicated method harking back to the encoder-decoder structure: they mix two pre-trained fashions — an encoder for the precise enter modality (like ViT for imaginative and prescient or an audio encoder for sound) and a Massive Language Mannequin (akin to LLaMA, Gemma, or Qwen). These fashions are related by projection layers that align their representations in a shared latent house, typically utilizing only a single linear layer. These projection layers study to transform the encoder’s output right into a format that matches the LLM’s anticipated enter dimensions and traits. The projection creates new embeddings/tokens from the enter modality that may then be injected into the LLM’s enter sequence. LLaVA is a first-rate instance of this structure for vision-language duties, whereas Spotify’s Llark and Qwen-Audio apply the identical precept utilizing audio encoders as a substitute of imaginative and prescient encoders.

Right here’s some pseudocode on how the fashions are stitched collectively:

# Extract options from ultimate layer of audio encoder
# Form: [batch_size, audio_seq_len, encoder_dim=1024]
audio_features = audio_model(audio_input)
# Venture audio options to match LLM's embedding dimension
# Form: [batch_size, audio_seq_len, llm_embed_dim=4096]
audio_embeddings = projection_layer(audio_features)
# Get textual content embeddings from LLM's embedding layer
# Form: [batch_size, text_seq_len, llm_embed_dim=4096]
text_embeddings = llm.embed_text(text_input)
# Concatenate alongside sequence size dimension
# Form: [batch_size, audio_seq_len + text_seq_len, llm_embed_dim=4096]
combined_input = concatenate([audio_embeddings, text_embeddings], dim=1)
# Feed them into the LLM as regular for technology
output = llm(combined_input)

Overview of structure

Llark makes use of OpenAI’s Jukebox and Qwen2-Audio makes use of OpenAI’s Whisper for the audio towers. Jukebox is a music technology mannequin however it will probably additionally soak up audio clips as enter and outputs a continuation of the audio clip. Whisper is used for transcribing voice to textual content.

Given their objective, the selection of audio module is obvious: Llark specialises in music evaluation, whereas Qwen2Audio primarily focuses on responding to voice directions with some primary audio and music evaluation capabilities.

Figuring out the optimum supply for extracting embeddings from giant pre-trained fashions includes analysis and experimentation. Moreover, deciding whether or not to fine-tune your complete module or freeze elements of it’s a essential design alternative. As an illustration, LlaVa’s coaching technique includes freezing the imaginative and prescient tower and specializing in fine-tuning the projection layer and language mannequin. We’ll go over this side of every mannequin under.

Llark: why Jukebox? Are these embeddings the perfect as of September 2024?

Figuring out the optimum location to extract embeddings from giant fashions sometimes requires intensive probing. This includes testing varied activations or extracted layers of the mannequin on totally different classification duties by a technique of trial and error. For music technology fashions, this might embrace duties like style recognition, instrument detection, emotion detection, in addition to evaluation of harmonic buildings and temporal patterns. Many business embedding fashions (like OpenAI’s embedding models) are educated particularly for embedding technology with specialised architectures and coaching targets, fairly than being fine-tuned variations of present language fashions.

The 2 largest publicly accessible music technology and music continuation (i.e.: in a position to soak up audio as enter) fashions are Jukebox and MusicGen. MusicGen is newer and sooner, and subsequently appeared like it will be the apparent option to me. Nonetheless, in accordance with this paper on probing MusicGen, embeddings extracted from Jukebox seem to outperform MusicGen on common in classification duties. The findings from this paper led to the authors of Llark utilizing the next method for extracting embeddings:

1. Embeddings are derived from the output of the thirty sixth layer of the Jukebox encoder following the method described in Castellon et al. (2021)
2. Unique Jukebox encoding:
   * 4800-dimensional vectors at 345Hz
   * For a 25s clip: over 4.14 * 10⁷ floating-point values
3. The authors use a downsampling method: Imply-pooling inside 100ms frames, leading to:
   * Downsampled frequency: 10Hz
   * Embedding dimension: 1.2 × 10⁶ for a 25s audio clip. Meaning a 2D array with form [240, 4800].
   * Retains temporal data (not like Castellon et al. who common over the time dimension)

(The downsampled embedding dimension is roughly 6x bigger than CLIP ViT-L14 fashions utilized in many multimodal imaginative and prescient fashions)

Qwen2Audio: Whisper

The embedding extraction for Qwen2Audio isn’t talked about intimately within the paper. Whisper is an encoder-decoder structure the place the encoder generates deeply discovered representations of the audio and the decoder decodes the representations to textual content (the transcription). In Qwen2Audio, it seems they extract embeddings from the ultimate layer of Whisper’s encoder, though they don’t point out whether or not they freeze it throughout coaching.

Pre-trained weights, coaching information and datasets

Sadly Spotify has not offered any datasets or their educated mannequin weights to the general public, noting:

“With respect to inputs: the inputs to our mannequin are public, open-source, Artistic Commons-licensed audio and related annotations. Nonetheless, every particular person audio file can have its personal, probably extra restrictive license. Lots of the audio recordsdata embrace “no derivatives” licenses. We encourage customers of the datasets to familiarize themselves with the restrictions of those licenses; in an effort to honor such licenses, we don’t launch any derivatives from the coaching information on this paper (together with query- response pairs or educated mannequin weights).”

They used the next datasets:

• MusicCaps (Agostinelli et al., 2023)
• YouTube8M-MusicTextClips (McKee et al., 2023)
• MusicNet (Thickstun et al., 2017)
• FMA (Defferrard et al., 2017)
• MTG-Jamendo (Bogdanov et al., 2019)
• MagnaTagATune (Legislation et al., 2009)

Llark particulars it’s coaching information technology course of within the following extract:

“We use variants of ChatGPT to extract the instruction- tuning information for all experiments. Nonetheless, the precise language mannequin used varies by dataset. We choose the OpenAI mannequin as follows: We use GPT-4 for all reasoning duties. We discovered that GPT-4 was rather more adept at following the complicated directions within the Reasoning process household. For datasets with greater than 25k samples, we restrict Reasoning information to a random subsample of 25k tracks.”

This leads to Q&An information like this:

The datasets used for coaching Qwen2Audio usually are not shared both, however the educated mannequin is extensively accessible and in addition is carried out within the transformers library:

For this mission, fine-tuning off a pre-trained Llark mannequin would have been optimum, given it’s reportedly good efficiency in opposition to the analysis benchmarks Spotify said within the paper.

Nonetheless, given they didn’t launch the weights for it, it’s unfeasible to start out coaching a mannequin like this from scratch with out a good bit of experience and cash. Spotify educated it on:

Our mannequin is educated on 4 80GB NVIDIA A100 GPUs. Coaching takes roughly 54 hours.

This could value round $700 utilizing a supplier like LambdaLabs.

Due to the above, I went with Qwen. Nonetheless, Qwen2-Audio doesn’t carry out that effectively throughout primary music duties like tempo and instrument detection. I element this under within the analysis part. Because of this the mannequin might be not giant sufficient or pre-trained sufficient to attain this process, however my hope is I might at the very least set a place to begin and framework for fine-tuning on this process sooner or later. As Alibaba state of their Qwen2-Audio blog post:

We additionally plan to construct bigger Qwen2-Audio fashions to discover the scaling legal guidelines of audio language fashions.

For my very own studying although, I did have a go at re-creating the mannequin utilizing torch and pre-trained fashions with the transformers library.

I additionally created datasets for Q&An information and embeddings. I generated quick type Q&An information for the URMP dataset, e.g.: “What’s the tempo of this monitor”, “What devices are taking part in on this audio”.

Here’s a notebook for operating Jukebox in a Colab setting to reap the benefits of a budget T4 GPU’s. I uploaded each Q&A and embeddings datasets to HuggingFace here.

Here’s a notebook with Llark replicated.

Transcription format

I selected ABC music notation because the output format that the language mannequin is predicted to transcribe the music in. Right here’s an instance of it:

X:1
M:4/4
L:1/16
Okay:none
Q:67
V:1 identify="Electrical Bass (finger)"
%%octave-default C4
GAA^2E3A2<A^2 | D^D^2E2A2A^4 A^2E2 | A2A^4A^2E2 A2A^4 | A^2E2A2A^4A^2E2A2 |
A^4 A^2E2 A2A^4A^2 E2 | A2A^4 |
V:2 identify="Brilliant Acoustic Piano"
%%octave-default C5
[E3C3][E3C3][E3C3] [E3C3][A^,2E2A^2] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3] |
V:3 identify="Electrical Guitar (jazz)"
%%octave-default C5
E'3C'3A^4E'3C'3 | A^4E'3 C'3A^4E'3C'3 | A^4 E'3C'3A^4 E'3C'3 | A^4E'3C'3A^4E'3C'3 |
A^4E'3C'3 A^4E'3C'3 | A^4 |

On this notation we have now the time signature and tempo outlined on the high denoted by ‘M’ and ‘Q’. The ‘L’ signifies the default word size of the notation, on this case a sixteenth word, which is the norm. We then outline every instrument and the default octave they need to adhere to when writing the notes for every of them. Right here’s a abstract of the important thing syntactical factors for writing notes in ABC music notation:

• Notes are represented by letters A-G, with lowercase letters indicating larger octaves
• Sharps are denoted by ^ earlier than the word, flats by _
• Pure indicators are represented by =
• Notice size is indicated by numbers after the word (C2 is twice so long as C)
• Dotted notes use a . after the word (C. is a dotted quarter word)
• Rests are represented by z, with numbers for length (z2 is a half relaxation)
• Chords are enclosed in sq. brackets [CEG]
• Ties are proven with a hyphen –
• Bar strains are represented by |
• Damaged rhythms use > or < between notes (C>D means dotted-C eighth word adopted by D sixteenth word)

Why ABC?

The explanations for selecting this notation are:

1. It’s a minimalist format for writing music
2. It’s extensively used and fashionable; language fashions have already got good comprehension of ABC notation as a result of intensive pre-training on it.
3. It’s versatile and might simply be prolonged to incorporate tempo adjustments, time signature adjustments, further taking part in kinds like talked about above, and so forth…

I transformed the MIDI recordsdata offered by the datasets to ABC notation utilizing this library. A pocket book for creating the datasets is here.

To judge each the unique mannequin and every stage of fine-tuning I carried out thereafter, I randomly chosen 30 samples of various complexity from the URMP dataset and ran the mannequin thrice on every pattern, manually analyzing all responses.

Via guide testing, I discovered the optimum decoding parameters to be a temperature of 0.7 and a top_p of 1.2. The utmost variety of tokens to return was capped at 2048. Adjusting the max appeared to have little distinction on efficiency.

The unique mannequin carried out poorly on this analysis set. Whereas it often predicted the tempo and devices appropriately, it principally failed to take action. A textual content file with the analysis outcomes is accessible here.

Given this place to begin, it’s unlikely that we’ll see sturdy outcomes from this experiment with out a sturdy pre-trained mannequin. Nonetheless, the objective is to develop methods that may be utilized sooner or later as extra superior pre-trained fashions develop into accessible.

I first tried fine-tuning with primary cross-entropy loss. Supervised fine-tuning with cross-entropy loss is a fast option to begin educating the mannequin however a primary loss perform like this has limitations as we are going to see under. The instinct behind this stage of coaching is that it will nudge the mannequin in the appropriate route and it will choose up any patterns or any customised ABC notation the dataset could have which the mannequin could not have seen earlier than.

Cross-entropy loss with instructor forcing

First, we educated it in a typical supervised fine-tuning method for language fashions. I used the SFTtrainer from the trl library for this, which makes use of cross-entropy loss with instructor forcing outlined step-by-step under:

1. The mannequin predicts the following token within the sequence.
2. The loss is calculated primarily based on the distinction between the anticipated chances (logits) and the precise subsequent token.
3. For the following prediction, the mannequin is given the precise right token (floor fact), fairly than its personal prediction. This is named instructor forcing, it helps stabilise coaching and considerably pace it up, particularly within the early levels.

The outcomes from this coaching section had been poor. It degraded the efficiency of the unique mannequin. The mannequin, which beforehand dealt with tempo and instrument recognition effectively, now principally obtained these incorrect. It additionally started producing garbled textual content output with infinite repetition. This occurred even when setting a low studying price, making use of gradient clipping, and utilizing low LoRA ranks to mitigate giant adjustments to the mannequin. Total, it appeared the mannequin was very delicate to the coaching utilized.

Nonetheless, whereas this coaching section could provide some enhancements, it gained’t result in optimum efficiency because of the limitations of our primary loss perform. This perform struggles to totally seize the mannequin’s efficiency nuances. For instance, when utilizing instructor forcing, instrument predictions can yield deceptively low loss throughout sure token sections. If an instrument identify begins with “V”, the mannequin would possibly confidently predict “Violin” or “Viola” primarily based on our dataset, no matter accuracy. Moreover, the loss perform could not precisely replicate near-misses, akin to predicting a tempo of 195 as a substitute of 200 — a small distinction that’s fairly correct however probably penalised closely depending on the distribution of chances amongst logits. It’s potential that neighbouring numbers even have excessive chances.

RLHF with PPO

Due to these limitations, we will create our personal customized loss perform that may extra precisely rating the response from the mannequin. That’s, given a predicted sequence from the mannequin, the loss perform might give it a rating between 0 and 1 on how good it’s.

Nonetheless, integrating this tradition loss perform into supervised fine-tuning presents a major problem. The difficulty stems from the non-linearity launched by the customized loss perform, which prevents the direct calculation of gradients. Let’s break this down:

In conventional SFT with cross-entropy loss:

• The mannequin outputs logits (uncooked scores) for every token in its vocabulary
• These logits instantly symbolize the mannequin’s prediction chances
• The loss perform compares these chances to the bottom fact
• Gradients might be computed instantly by this comparability
• The chain rule of calculus permits us to propagate these gradients again by the mannequin

With our customized loss perform:

• The mannequin should first generate full textual content output
• This technology course of includes sampling from chance distributions
• Our loss perform then analyses this textual content output (checking tempo, notes, and so forth.)
• This creates a non-differentiable step between the mannequin’s logits and our loss calculation
• The sampling and textual content evaluation steps break the gradient chain wanted for backpropagation

To beat this, reinforcement studying methods like Proximal Coverage Optimisation (PPO) might be employed. PPO is particularly designed to deal with non-differentiable loss capabilities and might optimise the mannequin by contemplating your complete coverage (the mannequin’s output distribution), fairly than counting on gradient data from logits.

Notice, there’s a lot of great articles on right here explaining PPO!

The important thing perception of PPO is that as a substitute of making an attempt to instantly backpropagate by the non-differentiable steps, it:

1. Treats the mannequin’s outputs as actions in a reinforcement studying framework
2. Makes use of the customized loss perform as a reward sign
3. Updates the mannequin’s coverage (its chance distributions over tokens) to maximise anticipated reward
4. Does this whereas making certain the up to date coverage doesn’t deviate too removed from the present one

This method permits us to successfully practice the mannequin with the customized loss perform, making certain efficiency enhancements with out disrupting the core coaching dynamics. The PPO algorithm’s conservative replace technique helps preserve stability throughout coaching, which is especially essential when working with giant language fashions.

Normally, this scoring perform can be carried out as a separate LLM within the type of a “reward mannequin” generally used when fine-tuning fashions through RLHF, which was a breakthrough first launched when ChatGPT got here out. Because of the nature of this process, we will manually write code to attain the responses, which makes use of fewer sources and is faster.

For time signature and tempo recognition that is simple to calculate. We extract all predicted gadgets with regex, for instance extracting the metre:

def extract_metre(self, abc_string):
return re.search(r'M:(S+)', abc_string).group(1)

The mannequin ought to study the syntax and construction we would like it to output within the SFT stage. If it outputs one thing that can trigger our regex to not discover something or error, we will simply skip that pattern, assuming it’s a small minority of the dataset.

We extract the anticipated tempo and write a perform that’s extra forgiving for small errors however penalises bigger errors extra closely:

• For small variations (≤10 BPM), it makes use of linear scaling.
• For bigger variations, it switches to exponential scaling.
• The ultimate loss is capped between 0 and 1.

Let’s break down the important thing parts of this tradition loss:

Code for the customized loss is here

1. Metre Loss

The metre loss focuses on the time signature of the piece. It compares the anticipated metre with the bottom fact, contemplating each the numerator and denominator individually, in addition to their ratio. This method permits for a nuanced analysis that may deal with varied time signatures precisely.

The metre loss makes use of a mix of linear and exponential scaling to penalise variations. Small discrepancies end in a linear improve in loss, whereas bigger variations result in an exponential improve, capped at a most worth of 1.

2. Tempo Loss

Tempo loss evaluates the accuracy of the anticipated beats per minute (BPM). Much like the metre loss, it makes use of a mix of linear and exponential scaling.

For small tempo variations (≤10 BPM), the perform applies linear scaling. Bigger variations set off exponential scaling, making certain that vital tempo mismatches are penalised extra closely.

3. Pitch Loss

The pitch loss is maybe probably the most essential part, because it assesses the accuracy of the transcribed notes. This perform makes use of the Levenshtein distance to match the sequence of notes in every voice.

The pitch loss calculation accounts for a number of voices, matching every predicted voice to the closest floor fact voice. This method permits for flexibility in voice ordering whereas nonetheless sustaining accuracy within the total pitch content material.

4. Instrument Loss

The instrument loss evaluates the accuracy of instrument choice for every voice.

This perform considers actual matches, devices from the identical household, and makes use of string similarity for extra nuanced comparisons. It supplies a complete evaluation of how effectively the mannequin identifies and assigns devices to every voice.

5. Combining the Losses

The ultimate loss is a weighted mixture of those particular person parts:

total_loss = (0.5 * pitch_loss +
0.15 * metre_loss +
0.15 * tempo_loss +
0.2 * instrument_loss)

This weighting scheme prioritises pitch accuracy whereas nonetheless contemplating different essential points of music transcription.

PPO coaching usually requires much more reminiscence than SFT for just a few causes:

1. A number of coverage evaluations — PPO wants to take care of each the present coverage (mannequin weights) and an “outdated” coverage to compute the chance ratio between them. This successfully doubles the mannequin parameters in reminiscence.
2. Expertise buffer — PPO shops a buffer of experiences (states, actions, rewards, and so forth.) to carry out updates in mini-batches. This buffer might be fairly giant and takes vital reminiscence.
3. Benefit estimation — Computing benefits requires retaining monitor of worth estimates and returns throughout trajectories, including one other layer of reminiscence overhead.
4. Extra optimisation targets — PPO tracks a number of loss parts (coverage loss, worth loss, entropy bonus) and their gradients, whereas SFT has a single loss.

Due to the above, we’re extra restricted than SFT within the dimension of the fashions we will practice and the way a lot it prices. Whereas the above coaching I might do on an A100 40GB in Colab, for the PPO coaching I wanted extra reminiscence. I educated on an H100 80GB, which might practice a LoRA with a rank of 128 and a batch dimension of 8.

My hyperparameter sweep was slender, I went with what appeared most intuitive utilizing batch sizes starting from 1 to 16 and studying charges from 2e-5 to 2e-4.

The mannequin made no enhancements to the duty. The textual content file with the outcomes is here.

I tracked varied coaching metrics utilizing Weights & Biases (WandB). Key metrics included the coverage loss, worth loss, whole loss, KL divergence, and the reward mannequin’s rating.

For all hyperparameter runs, the logs no enchancment within the rewards and loss calculated over time. The KL divergence remained inside the pre-defined threshold.