GreatScott! took a nearly twenty-year-old Nintendo Wii Remote, opened the shell, and fitted a Seeed Studio XIAO ESP32S3 Sense board inside so the controller could recognize what it was aimed at and switch appliances on or off. A small camera module sits at the front end of the remote after the original infrared sensor was moved aside with a longer ribbon cable.
The Home and A buttons are still attached to the new board, while a very small piezo buzzer has been inserted within the remote to provide feedback, and a li-po battery is providing all of the consistent juice you’d require in place of those old AA batteries. Once the plastic cover is closed again, the remote looks almost like it did before from the outside, but it’s been supplied with a camera, Wi-Fi, and a brain powerful enough to run a small neural network all by itself, so there’s no need to worry about sending all the data off to the cloud.
- HD Rumble 2
- Motion controls
- Built-in amiibo functionality*
Getting the network up and running began with over a thousand images of the target rooms and devices, as GreatScott! slung the camera at lamps, a TV, roller shutters, and a host of other appliances from a variety of distances, angles, and lighting conditions, then labeled all of the images, grouping them into seven categories, including an “unknown” class for anything he didn’t know what to do with.
Around 80% of those photographs were tossed into the training mix, with the remaining 20% serving as quality control to ensure that your machine was learning properly. After about fifty iterations of the training process, the best model they developed achieved 87.78% accuracy on images it had never seen before, which isn’t bad at all. They then compressed the network to fit inside the ESP32’s memory, reducing the file size to just 35 kilobytes. When you point the remote at a scene, the camera captures a 96 by 96 frame, which the network decodes roughly 1.4 seconds later and confirms with a short beep.
Control is then provided via MQTT messages over your house Wi-Fi network to a house Assistant server, which is already busy managing your Shelly smart relays and plugs. hitting the Home button initiates the scene-classification process; hitting the A button again instructs the model to turn on the appliance it believes you are pointing at. In testing, they were able to get it to correctly identify and switch on the gadget you wanted to control, but when you move things around in a real environment, it all falls apart.
Suddenly shifting a piece of furniture, changing the lighting, or pointing at two lamps that seem similar reduces the success rate to roughly two out of three attempts, which isn’t too awful but is a little hit and miss. This is because the model is classifying the entire scene, not just what it can see of any single object in that image, thus changing what’s in the background can sometimes completely throw it off.
This concept came after GreatScott! Take a look at the OpenMote project, where someone inserted an ESP32 board into a Wii Remote shell and mapped out which buttons do what. The problem was that you still had to know which buttons were for which things. Adding the camera and the scene classifier removes all of that, allowing the remote to determine what you’re pointing at based on the scene in front of it.
The end result is a fully functional, completely local, and pretty darn fun little prototype, though it’s still a little too slow and not quite accurate enough to replace every household remote, but who knows, maybe when the next generation of microcontrollers arrives with dedicated AI acceleration, everything will change. Until then, this customized Wii Remote demonstrates how to turn a simple point into a workable house command using only an old game controller, a few dollars worth of parts, and a very well trained model.
Source]
