Senior Machine Learning Engineer | ML Ops

github

youtube

Rhys Williams  • Peerlist

We are living in a very exciting in terms of the emergence of contrastitive language image pre-training and multi-modal extensions that are beginning to promise true zero-shot capability. CLIP, SigLIP and multi-stage pipelines such as Grounding DINO offer us some really exciting prospects, and we're already seeing CLIP's vision decoder married with cutting edge LLM's like <a target="_blank" rel="noopener noreferrer nofollow" class="text-green-text rich-link break-all link-break" href="https://huggingface.co/docs/transformers/en/model_doc/llava">Llava</a> and <a target="_blank" rel="noopener noreferrer nofollow" class="text-green-text rich-link break-all link-break" href="https://github.com/InternLM/InternLM-XComposer">InternLM-XComposer.</a>The project was borne out of two notions - excitement for the emergence of these foundation models and the fact the I find myself continually working with single board computers and SBCs in order to keep up with optimisation platforms like ONNX and TensorRT - platforms that will help us realise the world of true embodied AI we're all pushing for. So why not create and optimised vision foundation model swiss-army knife!Project overview:<ul><li>Full onxruntime support for CLIP and SigLIP with Grounding DINO around the corner.</li><li>Automatic model preprocessing and postprocessing switching.</li><li>Stripped of reliance on the Hugginface transformers, and tokenizers libraries in order to provide a lightweight set of dependencies. A bare metal implementation remains as a Huggingface lite nod if you will. </li><li>Concerted, pre-packaged inference session classes, with an incoming pip package. </li></ul>

Vision at a CLIP - Low-latency image rationalization

Delivered a technical demonstration and presentation about the various techniques available to deploy neural networks on embedded systems and edge devices in general.
• The demonstration had participants converting a myriad of multi-modal networks to intermediate representations with Intel's OpenVino, Nvidia's TensorRT and Open Neural Network Exchange.
• The presentation ended with a demonstration of the pratical implications of these toolsets with an AGX Xavier developer kit device influencing actuators in a robots head to track objects in realtime.

Presentation/ Technical Demonstration - Toronto Machine Lear

• Delivered a presentation and workshop regarding the power of GPU accelarated data science.
• Inroduced participants to RAPIDS - A growing ecosystem of libraries developed by NVIDIA to aid in the execution of end-to-end data science with hardware accelarators.