Vietnamese sung by AI: how many words survive?

The singing model behind MuziAI officially supports only English and Chinese. So what happens when you hand it Vietnamese, a language with six tones written as diacritics, where the same letters with a different mark make a different word? We measured how much of the lyric a speech recognizer can hear back and compared it with what was written. The answer: 78.6% to 94.7% of the lyrics come back with every diacritic correct, depending on genre, against 100% for an English control song. Most of what is lost is not words but marks, and according to our measurement log the main error is a slipped tone: strip the diacritics and the figures rise to 87.5% to 98.4%.

Three rulers broke before we got a number we trusted

We got this wrong three times before we had a usable measurement. Every time, the fault was in the ruler rather than the song, and that is the part most worth telling.

1. A fast speech recognizer, scored by character error rate over the whole song. Over instrumental passages it invented a line in Vietnamese, a video-channel-style “subscribe to our channel, don't miss our great videos”, and that invented text replaced the actual sung words instead of just being added. English was unaffected, so the comparison was rigged: an error rate of 0.012 for English versus 0.32 to 0.48 for Vietnamese.
2. Cutting the song into 12-second windows to make it easier to hear. Worse: almost every window produced the invented line, including passages where the whole-song pass had recognized the lyrics.
3. Separating the vocal from the backing first, then transcribing. Still invented text, while English still worked. At that point it was clear: the problem was the fast recognizer on Vietnamese, not the singing model.

The lesson: a Vietnamese error rate that has not been checked on cases with a known answer is not worth quoting. A lyric metric that hallucinates lyrics will unfairly punish exactly the language it hears worst.

The ruler that works: line coverage, tested on known answers

The final setup uses the larger recognizer, turns off its habit of conditioning each guess on the previous sentence (the mechanism that lets one invented line drag a chain of others after it), and adds thresholds that discard low-confidence segments. Instead of a whole-song error rate it scores line coverage: for each distinct lyric line (section labels removed, repeated lines counted once), find the best-matching stretch of the transcript, count what share of that line's syllables appear in order, then average across lines. Extra words are not penalized, so an invented line slipped in does not pull the score down.

Each song is scored twice: with diacritics (every mark must match) and without diacritics (all marks stripped, only the letter skeleton must match). The gap between the two columns is the “right word shape, wrong mark” share. Before using it, we ran the metric on four cases with known answers:

The third case is the one that matters: a transcript that lost every diacritic scores 0.19 with marks but 1.0 without. In other words the metric can tell “wrong word” apart from “right word, wrong mark”.

Results: pop, ballad, bolero and an English control

The ballad was the clearest, the bolero the least clear. In all three, the without-marks column is higher than the with-marks column: a meaningful share of words were sung with the right shape but the wrong mark. The bolero had both the biggest gap (0.089) and the lowest without-marks score (0.875), so in that song the model both slipped more marks and dropped more words. We have not measured why. Tone slips matter because they change the meaning without changing the letters: ma, má, mà, mả, mã and mạ are six different words.

The same pattern in the sing-over-a-beat tool

The site's tool for singing lyrics over an existing beat runs on a different singing system and is scored with a recognizer built specifically for Vietnamese, so its numbers are not directly comparable with the table above. The pattern repeats, though:

The pop track reaches 1.000 without marks: every word was sung, and the only loss was diacritics. And on the ballad, the same metric scores the original human vocal over that beat at 0.979, so even a real singer does not hit a perfect 1. Part of every score is simply the limit of the listener.

Limits and what we don't know yet

• The main table covers only three Vietnamese songs (pop, ballad, bolero). Enough to see the pattern, not enough to rank genres with confidence; we will re-measure as the song library grows.
• It measures what a machine hears, not what a person hears. We have not run a listening test with real people, so we do not know how many tone slips a human ear repairs from context.
• Extra words are not penalized. That is a deliberate choice to make the metric immune to invented lines, but it also means that if the singer adds words that are not in the lyrics, the metric will not catch it.
• The without-marks column strips vowel marks too (ă, â, ê, ô, ơ, ư, đ), not just tone marks. The gap between columns is “wrong diacritic” in general; we do not yet have a column that isolates tone alone.
• A ruler that works on one kind of audio may fail on another. The same larger recognizer, listening to vocals that had gone through the site's voice-swap tool, invented the very same “subscribe” line on 3 out of 3 files. There we had to switch to a Vietnamese-specific recognizer.

How to write Vietnamese lyrics that survive the singer

1. Pick the genre by how much the words matter. If the lyric is the point (a dedication, a story song, a jingle with a shop name in it), ballad (0.947) and pop (0.909) kept the words better than bolero (0.786) in our tests.
2. One complete sung phrase per line, with section labels. Put [Verse] and [Chorus] on their own lines and break the line after every phrase. In the sing-over-a-beat tool, lyrics pasted as one solid block made the system treat the whole song as one section; it came in 16 seconds late and sang like speech.
3. Check the words where a tone change flips the meaning, in the places that matter. The chorus line, a person's name, a shop's name are where one slipped tone hurts most. Listen to exactly those words in the finished song.
4. Generate a few versions and keep the clearest. Every generation is a fresh random draw, so it is worth retrying before rewriting the lyrics.
5. Do not grade the result with an ordinary speech-to-text app. Our first three rulers all invented lyrics on sung Vietnamese. If a transcript shows something odd like “subscribe cho kênh…”, the transcriber is at fault, not the song.