It’s no secret that device detection is challenging. In the machine learning world, the common analogy for what we do is called the “cocktail party problem.” No, that doesn’t mean we sit here in our bespoke suits and satin gowns and sip martinis while we work (alright, sometimes…). The cocktail party problem refers to the challenge of training a machine to pick out a particular voice in a room — or cocktail party — of people speaking. That’s a pretty easy feat for us humans, but quite a task for a machine.
Luckily, in speech recognition circles, it has mostly been solved (though some situations remain quite difficult). When you ask your voice assistant what the weather is going to be tomorrow, while your kids are chatting nearby, you’ll likely still get the weather report. But have you ever tried asking for the weather while continually changing your voice (think pitch, accent, speed). Much more challenging! At the risk of oversimplifying things, that’s essentially what Sense is doing. Your home is full of devices that “speak” very differently, all yelling over one another. Like a cocktail party met a frat party met a party of screaming toddlers. Throw in a dash of people not even speaking the same language and then a dash of people who talk incredibly slowly (think Ents, for you Lord of the Rings fans), and you have your house at the electrical level.
This isn’t a perfect analogy. The truth is a good deal more complicated. Electrical devices don’t share the same magnitude range as human voices; they’re much broader. At a party, you might have someone talking at a whisper and then someone screaming. Home electronic devices range from around .5W to 10kW (almost a 2 million percent increase in magnitude). Making a messy comparison to the volume of a human voice, if you heard a sound 2 million times louder than a whisper, you’d be dead. A jet engine is only about 4,000 times louder than a whisper. Let that sink in for a minute.
And then there’s the matter of time scales. Most people talk at roughly the same speed, between 120 and 150 words per minute. But the devices in a home cycle at radically different rates, from millions of times every second (like the switching power supplies that power many consumer electronics) to having a period that’d be better measured on an hourly scale (like electric vehicles).
With this in mind, the prospect of “training” Sense should look a lot more problematic. Sense needs to see repeated patterns, enough repeated patterns that it can consistently detect a device regardless of power line noise or any other devices running concurrently (which make the “voice” sound different). Exactly how many noise-altered iterations of a device Sense needs to see varies, but it’s much more than you would be able to comfortably tag. This explanation is avoiding the topic of resolution, where Sense looks at sub-second features like the on/off transients of a device to properly identify it. The resolution of the Power Meter is downsized to allow for a better viewing experience, but even if we showed you the full data at a 1MHz resolution (our engineers are probably cringing while reading this), it would be near impossible to manually mark the exact beginning and end of a millisecond event with any accuracy — and Sense needs accuracy. To complicate things even further, the signatures of some devices, like LEDs, can change after being turned on/off in succession. And if you want another complicating factor: Manually turning off a device via its power switch or circuit breaker can look quite different than a “natural” device cycle. It’s a tough nut to crack — a nut wrapped in an inch-thick shroud of steel. But, never say never! The more data that Sense gathers, especially ground truth data from your smart plug devices, the closer we get to a feasible method of manually teaching Sense.
How you can help now
In the meantime, you can still help Sense learn. We have introduced a variety of features that take advantage of user input to help improve device detection in your home and for the entire user base.
- Network Identification allows Sense to see some of the simple “handshake” messages put out by your networked devices.
- Integrations with smart bulbs from Philips Hue and smart plugs from TP-Link Kasa and Belkin Wemo will net you instant detections for connected devices and provides great data to the Data Science team.
- If you have an Ecobee smart thermostat, be sure to turn on the historical Ecobee integration under Settings > My Home > Connected Devices > Data Sources.
- Renaming your devices, supplying the make/model, taking advantage of the Community Names feature, and filling out your Home Details feeds the Data Science team great data that improves detection for everybody.
- When Sense finds a device, but you’re finding the detections to be inaccurate, you can report it as “not on.” This feeds our Data Science team valuable information so they can continue to refine the detection model.
Remember, even without native detections, you can still take advantage of Sense insights. The Power Meter is a fantastic tool that provides a real-time view of your energy consumption. Try turning on and off your devices while watching in the Power Meter, to identify how much they consume. You can do the same for the “Always On” devices in your home, identifying how much they’re costing you every day.
While the challenge is considerable, we’re constantly making great progress. With your help, Sense is getting better all the time. Check out this post to learn what strides we've made in 2019.