Ask HN: How are these AI cover songs made?
There's been a recent flurry of AI covers on Youtube and Instagram. For example: https://youtube.com/@fakemusicbr?si=G-RzKcm2iuXX56kI
Some of these songs are _really_ good. And pretty far ahead of where I thought audio AI was. Does anyone know what models and tools are used? I imagine once you have stems you can maybe style / genre transfer and use a reference audio file perhaps? Like image and video models.
Style transfer for music has actually made huge strides in the past few years.
When I composed this piece of classical music over nearly a decade ago, it was originally intended for strings. A few months ago I tried using GenAI to do a cover in a chamber quartet style by providing a WAV file of the original piano performance and it stayed almost 1:1 faithful to the original in terms of motifs and structure.
Related blog post with comparisons of the original piano performance and the cover for reference.
https://mordenstar.com/blog/screwdriver-sonata
i think most of these are suno.com covers, to bypass copyright you can use phonetic spelling for the lyrics.
Does it flag for copyright? I normally just copy-paste the lyrics into Suno, just to see how it would make the songs. They don't sound the same so I'm guessing they don't have that many songs in their copyright checks.
They definitely scan uploaded audio files for copyright. It's not comprehensive and I'm sure people work around it by slowing songs down, altering pitches, etc. (the same way people used to get around copyright detection on Youtube).
Most of these no doubt are just websites, you can search for 'ai song generator'.
Where you can input your lyrics, styles, etc. You can create all original songs by having AI generate the lyrics even.
>Does anyone know what models and tools are used?
One of the best in my opinion: https://huggingface.co/tencent/SongGeneration
One of the most popular: https://huggingface.co/facebook/musicgen-medium
The "transformers" is just a python library and you just add all the parameters in a constructor:
processor( text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"], padding=True, return_tensors="pt", )
This one can run on like 16GB of vram and modern cards produce them trivially.
>I imagine once you have stems you can maybe style / genre transfer and use a reference audio file perhaps?
You can do audio to audio; but it's not as easy as just generating the whole thing from text.
Appreciate your response, thank you.
I had a look at what you provided. The "ai song generator" websites and these models aren't able to genre transfer with the exceptional quality of the linked youtube channel. These have incredibly good alignment to the original songs, but the genre totally transformed. For example, the Nirvana one on that channel. Song structure is entirely preserved, as well as melodic and harmonic content. But the genre is transformed to 80s synth pop. It's the same for any recent upload to that channel.