AI Takes a Bow: Interactive GLaDOS Robot Among 9 Winners in Hackster.io Challenge

YouTube robotics influencer Dave Niewinski has developed robots for all the things from driveable La-Z-Boy chairs to an AI-guided cornhole tosser and horse-drawn chariot racing.

His current Interactive Animatronic GLaDOS venture was amongst 9 winners within the Hackster AI Innovation Challenge. About 100 contestants vied for prizes from NVIDIA and Sparkfun by creating open-source initiatives to advance the usage of AI in edge computing, robotics and IoT.

Niewinski received first place within the generative AI purposes class for his revolutionary robotic primarily based on the GLaDOS information from recreation sequence Portal, the first-person puzzle platform from online game developer Valve.

Different prime winners included contestants Andrei Ciobanu and Allen Tao, who took first prize within the generative AI fashions for the sting and AI on the edge purposes classes, respectively. Ciobanu used generative AI to assist nearly strive on garments, whereas Tao developed a ROS-based robotic to map the within of a house to assist discover issues.

Harnessing LLMs for Robots

Niewinski builds customized purposes for robotics at his Armoury Labs enterprise in Waterloo, Ontario, Canada, the place he makes use of the NVIDIA Jetson platform for edge AI and robotics, creating open-source tutorials and YouTube videos following his experiences.

He constructed his interactive GLaDOS robotic to create a private assistant for himself within the lab. It handles queries utilizing Transformer-based speech recognition, text-to-speech, and enormous language fashions (LLMs) operating onboard an NVIDIA Jetson AGX Orin, which interfaces with a robotic arm and digital camera for interactions.

GLaDOS can monitor his whereabouts within the lab, transfer in numerous instructions to face him and reply shortly to queries.

“I like doing issues with robots that individuals will take a look at and say it’s not what that they had instantly anticipated,” he mentioned.

He wished the assistant to sound like the unique GLaDOS from Portal and reply shortly. Fortuitously, the gaming firm Valve has put all the voice strains from Portal and Portal 2 on its web site, permitting Niewinski to obtain the audio to assist practice a mannequin.

“Utilizing Jetson, your common question-and-answer stuff runs fairly fast for speech,” he mentioned.

Niewinski used NVIDIA’s open-source NeMo toolkit to fine-tune a voice for GLaDOS, coaching a spectrogram generator community known as FastPitch and HiFiGAN vocoder community to refine the audio high quality.

Each networks are deployed on Orin with NVIDIA Riva to allow speech recognition and synthesis that’s been optimized to run at many instances the real-time price of speech, in order that it may well run alongside the LLM whereas sustaining a clean, interactive supply.

For producing real looking responses from GLaDOS, Niewinski makes use of a domestically hosted LLM known as OpenChat that he runs in Docker from jetson-containers, saying that it was a drop-in substitute for OpenAI’s API. All of this AI is operating on the Jetson module, utilizing the most recent open-source ML software program stack constructed with CUDA and JetPack.

To allow GLaDOS to maneuver, Niewinski developed the interactions for a Unitree Z1 robotic arm. It has a stereo digital camera and fashions for seeing and monitoring a human talking and a 3D-printed GLaDOS head and physique shell across the arm.

Attempting on Generative AI for Trend Match

Winner Ciobanu, primarily based in Romania, aimed to enhance the digital clothes try-on expertise with the assistance of generative AI, taking a prime prize for his EdgeStyle: Fashion Preview at the Edge.

He used AI fashions corresponding to YOLOv5, SAM and OpenPose to extract and refine information from photos and movies. Then he used Steady Diffusion to generate the photographs, which he mentioned was key to attaining correct digital try-ons.

This technique taught the mannequin how garments match totally different poses on individuals, which he mentioned enhanced the realism of the try-ons.

“It’s fairly useful because it permits customers to see how garments would look on them with out really making an attempt them on,” mentioned Ciobanu.

The NVIDIA JetPack SDK supplied all of the instruments wanted to run AI fashions easily on the Jetson Orin, he mentioned.

“It’s super-helpful to have a steady set of instruments, particularly while you’re coping with AI tech that retains altering,” mentioned Ciobanu. “It actually lower down on the time and trouble for us builders, letting us focus extra on the cool stuff we’re constructing as an alternative of getting caught on tech points.”

 Discovering Misplaced Objects With Robotic Help

Winner Tao, primarily based in Ontario, Canada, created a robotic to minimize the burden of looking for issues misplaced round the home. His An Eye for an Item venture took prime honors on the Hackster problem.

“Discovering misplaced objects is a chore, and up to date developments in zero-shot object detection and LLMs make it possible for a pc to detect arbitrary objects for us primarily based on textual or pictorial descriptions, presenting a possibility for automation,” mentioned Tao.

Tao mentioned he wanted robotic computing capabilities to catalog objects in any unstructured surroundings — whether or not a lounge or massive warehouse. And he wanted it to additionally carry out real-time calculations for localization to assist with navigation, in addition to operating inference on bigger object detection fashions.

“Jetson Orin was an ideal match, supporting all performance from textual content and picture queries into NanoDB, to real-time odometry suggestions, together with leveraging Isaac ROS’ hardware-accelerated AprilTag detections for drift correction,” he mentioned.

Different winners of the AI Innovation Problem embody:

• George Profenza, Escalator individuals tracker, 2nd place, Generative AI Purposes class
• Dimiter Kendri, Cooking meals with a neighborhood AI assistant utilizing Jetson AGX Orin, third place, Generative AI Purposes class
• Vy Phan, ClearWaters Underwater Picture Enhancement with Generative AI, 2nd place, Generative AI Fashions class
• Huy Mai, Realtime Language Phase Something on Jetson Orin, 2nd place, Generative AI Fashions class
• Fakhrur Razi, Autonomous Clever Robotic Procuring Cart, 2nd place, AI on the Edge Open class
• Workforce Kinetika, Counting for Inspection and High quality Management with TensorRT, third place, AI on the Edge Open class

Be taught extra about NVIDIA Jetson Orin for robotics and edge AI purposes. Get began creating your individual initiatives on the Jetson AI Lab.