Highly enthusiastic engineer hoping to pay it forward to society through product innovation and teaching

projects

devto

medium

github

Samyuktha Ganeshkumar  • Peerlist

Video2Music is a novel AI-powered tool that can generate music for any short video of max 30 seconds. It is hosted and deployed on Replit, making it easy for anyone to use.Video2Music works by first extracting images from the video every 2 seconds. It then uses InstructBLIP to identify what is happening in each image. Finally, it uses the Google Text-Bison (Replit ModelFarm) model to generate an overall description of the video.This information is then used to prompt the same model to generate a music prompt of instruments, genre, mood, feel, time signature, BPM, kbps, and kHz suitable for the video. This prompt is then used to generate music for the entire video length using the MusicGen Facebook model.Video2Music also includes functionality to alter the video description and music prompt generated so that the user can generate the music to their liking.The tool can be most helpful for content creators or anyone who wants a soundtrack perfectly synchronized to the content and mood of their short videos!Note: The audio cannot be used for commercial purposes since the MusicGen model has weights licensed as CC-BY-NC 4.0 (NC = non-commercial)PS. The deployed demo URL seems to be a bit slow while generating. The Replit itself will be faster.