Building multimodal LLMs

Kunal Dhawan  • Peerlist

>Built from scratch an ASR system in my mother tongue, Hindi using the toolkits: HTK and Kaldi >Coded various acoustic models- Monophone-HMM system, Triphone-HMM system, DNN-HMM system and used a bi-gram language model.Performed the steps: Data Collection(150 sentences*7 speakers), Lexicon Building, Feature Extraction, HMM parameters initialization, Baum-Welch re-estimation, fixing the silence model, realigning training data, graph building and decoding. Obtained Word Error Rate of 2.39% >Keywords: Automatic Speech Recognition, Kaldi, HTK, HMM, DNN, GMM

Automatic Speech Recognition System for Hindi

>Developed a system to detect 7 emotions (Anger, Fear, Happiness, Sadness, Disgust, Neutral, Surprise) from Human Speech Samples (using Surrey Audio-Visual Expressed Emotion (SAVEE) Database). >Concepts utilized: explored first central Spectral Moment time-frequency distribution augmented by low order Cepstral coefficients for feature extraction, modeled the data using GMM and adapted it using Maximum Aposteriori (MAP) algorithm, trained a multi-class SVM classifier with Gaussian kernel for the purpose of classification >Keywords: Speech Signal Processing, Emotion Detection, MATLAB, GMM, UBM, SVM

Emotion Detection from Speech

Bachelor Thesis Project (Advisor: prof. Rohit Sinha) -In-depth literature review and implementation of essential E2E ASR components: Connectionist Temporal Classification (CTC), Recurrent Neural Network Transducer (RNN-T), Attention based Encoder-Decoder Model, Raw Waveform input based CNN ASR systems. -The aim of this project is two-fold: 1)Understand/Interpret what the layers of the RNN/LSTM based encoder system are trying to learn so as to build more intuitive End-to-End ASR models 2)Build robust ASR systems for 'Low Resource' Indian Languages Keywords: End-to-End ASR, CTC, RNN-T, Attention based sequence models, Low-Resource ASR

End-to-End Speech Recognition

Designed a differential drive robot capable of indoor autonomous navigation at Robotics Club, IIT Guwahati using Open-Source platform ROS. Achieved 3D mapping of the surrounding (using Visual Odometry on data obtained from Kinect 360 ) and hence implemented concept of SLAM (Simultaneous Localization and Mapping ) using it

FAIR-First Autonomous Intelligent Robot

Built a framework for the popular Texas Instruments Ultra Low Power Microcontroller- MSP430.This framework is like a shield for the microcontroller. This framework provides all major peripherals for any microcontroller at one place so that it becomes easy for a person to build small projects as he has access to the various IO devices and is the ideal learning platform for beginners. I have used the TSSOP 28 pin MSP430G25553 as the microcontroller and have following peripherals on the board - Mic , Navigation Switches , LDR , 16*2 LCD Display , Capacitive touch , 6 Charlieplexed LEDs , Buzzer , RGB LEDs (their intensity is controlled using PWM) , Pins for UART , SPI and I2C communication in addition to the JTAG programming Pins. The board also features a stable power supply due to the usage of 3.3V LDO and an amplifier circuit for the analog Mic. The schematic and Board Design was made by me from scratch on Eagle and then I also fabricated the PCB myself using the Tonner Transfer method in the TICEPD(Texas Instruments Center for embedded product Design)  lab. After the fabrication, I also individually programmed and tested each of the peripherals. This project served as a great learning Experience for me.

Framework for MSP430

Made an Intelligent Robot , which can shoot the ball into the Goal Post of other Player using concepts of Computer Vision .Used Raspberry Pi to operate the bot(wirelessly) and OpenCV for processing the Video input. The implemented Code helps the bot to detect the ball in the environment, move towards it (2-D motion), align in correct position and score a goal in presence of opponent team bot.

Kick Ball Theory

In this project, a two channel heart sound acquisition circuit has been designed stage by stage and a consolidated working model has been implemented (the two channels are used to simultaneously read heart sound data from two different valves). This data is parallel displayed on oscilloscope to study the temporal variation of the heart sound and also fed to a speaker system to aid interpretability of the signal being displayed. Further the signal obtained is discretized and fed to a compute system where a GUI application is developed to give additional information about the heart sound acquired, like the heart rate of the person being studied and also providing frequency domain analysis of the signal obtained.

​Multichannel Heart Sound Signal Acquisition 

Multichannel Heart Sound Signal Acquisition a

>Developed a system for classification of musical instruments belonging to different instrument families exploiting various temporal and spectral features that characterize the 'timbre' of a particular instrument >Used a Hierarchical framework with total 5 classifiers and designed extraction algorithms for following features: Spectral Centroid, Roll-Off, Crest, Flatness, Slope, Skewness, Zero Crossing Rate, Variance of Spectral Centroid, Fundamental Frequency, MFCC and 4 more. Used basic Neural Networks as classifiers and achieved more than 96% accuracy for each class on the Iowa Database. >Keywords: Audio Signal Processing, Neural Network, MATLAB, temporal and spectral features

Musical Instrument Detection

Developed a bot that could accurately follow a human in a given closed enclosure. Concepts used were: Image processing: to detect the human and to calculate the change in position of the subject , PID (Proportional , Integral , Derivative ) : to minimize the error during motion of the bot and to ensure smooth motion even in case of sudden changes in subject’s location , ROS : to communicate between the python script which implemented image processing and the microcontroller(Arduino) which was responsible for bot motion .

Person Follower Bot

Built a bot which autonomously catches a ball falling down an inclined plane having obstacles. Detailed description of project environment and bot working can be accessed from my personal webpage . The concepts used were : Tracking the position of ball and finding obstacles using Image processing , doing a linear curve fitting for approximate motion of ball using Linear regression, sending data to bot for autonomous movement using subscribers and publishers in ROS(Robot operating system).

Phototron-Autonomous Ball Catching Bot

Built an interactive online IoT application which enables user to book seats of a bus from their homes . A remote web server hosts the data of layout of bus seats and their occupancy in real time .Data is read from various sensors placed on the seat and sent to web server(where data is managed) via Bolt (IoT device) connected to microcontroller (Arduino).

Smart Bus Booking System

Developed with the aim of providing freedom of movement to any individual irrespective of disability. The speed of the wheelchair is controlled by the normal force exerted by the person against it’s backrest which is measured by a force sensor placed there and the direction is controlled by angular motion of user’s head with respect to vertical measured by a gyrometer placed on a cap worn by the user.

Smart Wheelchair for Differently Abled

This project focused on the design of a 8 bit (1 byte) Static Random Access Memory having serial input facility and also a differential voltage sense amplifier for noise robust read operation of the memory designed, using 180nm CMOS technology(Software used - pyxis from Mentor Graphics)