Why AI covers sound raspy, and how we measured it

When you turn a song into your own voice (an AI cover), the result sometimes sounds raspy, crackly, chopped into little pieces. We chased this through many rounds of testing and broke at least nine of our own measuring sticks along the way. The short answer: the crackle that survives every fix lives in the voice sample you upload. The converter copies its roughness, filtering cannot remove it, and a better sample is the only cure.

The first measurement said “off-key”, and it was wrong

The pipeline has three steps: a vocal separator pulls the singing out of the mix, a voice converter swaps in the timbre of your sample, and the new vocal goes back over the original backing track. Most of the measurements below were taken on the earlier zero-shot converter, which needed no training; for “my own voice” the site now trains a small temporary voice model for each job and deletes it within 24 hours.

Our first check was the pitch correlation between the original vocal and the new one: 0.722, which reads like “the machine sings wrong notes”. Looking closer, the median deviation was 0.00 semitones, the MAD 0.10, and 33 of 1,197 frames (2.8%) had jumped exactly one octave, a classic pitch-tracker error rather than a singing error. Drop those frames and the correlation is 0.994. Since then we report pitch as median deviation, MAD and the share of frames within ±1 semitone (92.6% on the first test song).

Lesson two: the converter is random. The same settings run five times swing across a range of 0.064, larger than the 0.046 gap between the five candidate voices we were comparing. So any comparison we draw conclusions from now runs at least five times per branch. The “σ” figures below are differences in standard deviations.

What “raspy” looks like when you measure it

You could hear the chopping, yet three earlier measures all said “nothing wrong”: 0.0% of frames lost voice, amplitude depth 3.17 dB (a human singer: 3.13), no pitch jumps. The sound was loud enough and on pitch, but the harmonics had broken up into noise. Periodicity catches that:

Part of it is hiss: the waveform generator adds 5 to 7 dB above 8 kHz. On a source cut off at 16 kHz, the output's 16 to 20 kHz band sat at −23.2 dB, against −38.7 for a real human voice: 100% invented, and removable.

The crackle, though, stayed. Phonetics measures a rough, gravelly voice with jitter (how much consecutive vocal-fold cycles differ in length) and shimmer (how much they differ in amplitude). The normal-voice threshold in the analysis tool we use is 1.04% jitter. None of our earlier measures could see it.

Does a raspy sample carry over into the cover?

We first said yes, then took it back. A phone recording by the site's own builder had 2.8 times the envelope flutter of a clean sung sample, yet with the song fixed and only the sample swapped, the envelope measure showed the output just 10% worse. That measure tracks loudness wobble, not cycle-to-cycle variation.

Measured with jitter, the picture flipped: the clean sample gave an output of 0.92%, the phone sample 1.67% (+81%, 18.3σ). The phone sample itself had 2.27% jitter and 11.0% shimmer, baked into the recording (half-spoken delivery, gravelly phrase endings, a phone mic). Compression was not to blame: 30 kbps encoding moved a clean sample only from 1.03% to 1.10%. Neither was separation (+4% flutter, 1.3σ) or reverb (+5%, 2.3σ). The converter copies the voice quality of the sample.

The same day, two other conclusions went wrong the same way. A guidance setting went to 0 for every sample because it cut hiss by 43% on a clean sample, yet it raised jitter by 15% (6.5σ). A pitch-gap patch was rejected because the dropout measure called it noise, while jitter showed it winning by 3.7σ. The rule: a measure built for problem A has no right to judge a fix for problem B.

What shipped: build a 24-second sample from the least raspy one-second pieces (output 1.718 → 1.580%, 3.6σ), patch pitch gaps, and use guidance 0.7 for user samples. A full run with the phone sample went from 1.50% to 1.26% jitter, shimmer 7.89 to 5.29, HNR 17.8 to 19.4 dB, at the cost of a conversion step about 50% slower.

And here is the ceiling. That sample had only 29 seconds of singing and the pipeline needs 24, so the picker could drop only the 5 worst pieces (2.14 → 1.96%). Its best piece still measured 1.37%, rougher than the median clean singing voice (0.94%). The machine cannot invent clarity the sample never had. That is why the site now measures sample roughness on upload and warns from 1.5%: the phone sample reported 2.13% about 11 seconds in, while a clean 24-second sung sample reported 0.96% and triggered no warning.

12 seconds or 24 seconds of sample?

The site originally trimmed samples to 12 seconds because longer samples are slower: on the same 247-second song, a 20-second sample took 253 seconds to convert, a trimmed 12-second one 162 seconds (−36%). The claim that “a longer sample is not more similar” was inferred from the architecture and never measured. So we measured it, five runs per branch, scoring similarity on a stretch of the singer's voice that no sample contained (0.750 means unconverted, 0.866 is the same person in a different passage):

Every 24-second window beat every 12-second window on similarity, and the start point matters as much as the length. On the phone sample, 12 to 24 seconds lifted similarity from 0.684 to 0.747 (6.8σ) while the crackle stayed put. A longer sample fixes “does it sound like me”, not the crackle. A sample-scoring heuristic we had built ranked the 24-second windows in reverse order of the real results, so we threw it out.

Limits and open questions

• Vietnamese comes out about three times worse than English: 2.9% of frames lose voice versus 1.0% (noise range 0.7 percentage points). Consonants get swapped: «Gà gáy vang trời» became «cáo kẽ vang trời», «hỏi» became «khỏi». Not fixed yet.
• Filtering afterwards does not help: denoising the output made flatness worse, and karaoke separation plus de-reverb cut similarity by 0.028. The roughness sits inside the harmonics the machine generates, not on top of them.
• The source song matters too: the output is 2 to 3.6 times dirtier than its source. A source with flatness 0.0047 comes out around 0.011; a rip cut at 16 kHz (0.0070) comes out around 0.025.
• The 1.5% warning threshold rests on three data points (2.14%, 0.94 to 1.03%, 0.79%). The browser recorder turns off echo cancellation, noise suppression and auto gain, which is standard practice but untested with a real microphone.
• Per-voice fine-tuning of that converter raised a similarity score from 0.64 to 0.73 in one test, but that score came from the converter's own voice encoder, which flatters its own output. A later check with an independent voice-comparison model showed no gain (0.647 for both). The site has since moved to a different per-voice approach, so the zero-shot numbers in this post describe the earlier pipeline.

How to use this when you make a cover

1. Sing, don't talk. Half-spoken lines and gravelly phrase endings get copied straight into the output.
2. Record 20 to 40 seconds of real singing somewhere quiet. The site uses 24 seconds of singing; under 15 seconds triggers a warning.
3. Watch for the warning box a few seconds after upload. At 1.5% or above, re-record the sample.
4. Use a high-quality song file. A rip cut off at 16 kHz gives you a backing track cut off there too; the site tells you when the source is below 17 kHz.
5. Pick a sample close to the song's range. Dropping an octave (female song, male sample), measured in a single run, gave 1.28% jitter versus 0.75% when keeping the key, but similarity to the sample was 0.74 versus 0.52.
6. For songs in Vietnamese, listen back for swapped consonants such as h/kh and g/c.