Featured Image for How Sense learns about your devices
How Sense learns about your devices
Data Science
Cui-Lyn Huang

One of the advantages of Sense is that it detects a broad set of household devices without the need for you to train it. By taking 4 million recordings per second, Sense builds up a very detailed library of device signatures over time that let it recognize which devices are on in the home and when. Many of you have asked us if you can help speed up this process by manually identifying devices. Here, we’ll explain the data science behind our approach, and why training needs to be driven by data first and foremost. But fear not — you can help train Sense by renaming “unnamed devices” and sending us feedback when we get events wrong.

Context provides important clues

In order to accurately identify a device, Sense needs to see repeated occurrences of the device in its typical operating context. From the standpoint of an electrical signature, many devices are not simply on or off — rather, they operate in a range of different states. Your typical dishwasher could be on, and the motor just kicked on; or it could be in rinse, drain, or standby mode. Each of these states looks very different to Sense. Seeing a device’s full cycle and varied states, helps our data science team identify it. Sense also uses its knowledge of other identified devices to determine which device is which. For example, your dryer is likely to run after your washer. Sense data scientists look for these sorts of contextual time patterns to help identify your devices. Finally, the real-world operation of some devices contains useful clues that may be absent during an artificial training process. For example, microwave ovens usually are activated for multiples of 30 seconds, but a purposeful “training session” would probably run the oven for much shorter bursts.

More time = more data = better detection

Sense needs to see many occurrences of a variety of your home appliances turning on and off before it begins to distinguish them as unique items. This is because Sense is learning your home. In technical terms, Sense uses unsupervised machine learning to detect your devices. The “unsupervised” refers to the fact that there is no “ground truth,” because there are no alternative methods to definitively determine each device that is on or off in your home. Sure, you might be able to tell Sense with certainty when your toaster is on, but doing this for every single device — and all the states of those devices — would be really time consuming and boring!

Sense is like cat videos. No really.

To provide a related example, scientists at Google X trained an “artificial brain” to identify cat videos on YouTube by feeding it over 10 million videos. Instead of conducting pattern matching against videos labeled with specific features such as “human”, “cat,” or “cars,” — referred to as the training set — the system classified the images based on the patterns it found in the data alone. Again, it is important to note that Google X did this by using the images in their natural context. The process would simply not work if, say, a set of cat drawings had been used for that goal. To get a feeling for the concept of unsupervised learning, imagine you’re a little kid helping your family with the laundry and you’re surrounded by an assortment of socks. You don’t need to know your colors to begin to group blue socks with blue, white socks with white, and so on. Although device detection is considerably more difficult than sorting laundry, thankfully we don’t need quite as much data as Google in order to train Sense!

When you consider the number of household devices, the many states of each device, and the variation between devices of a single type; it starts to become clear why Sense takes time to learn.

Well then what can I do in the meantime?

In the meantime, you can still help improve Sense by naming the “unknown” devices we detect, and fixing labels when we have incorrectly identified a device. We can already see large benefits from such feedback, and as Sense gets deployed in many more homes such misidentifications will become increasingly rare!

To identify an Unnamed Device, you’ll first need to figure out what it is. To dig up clues on unnamed devices, navigate to the Device screen where you’ll be able to add them to your Timeline, enable alerts, and get historical information about their usage. Once you’ve figured it out, you can rename the devices on the Device screen and help the Sense community. Re-naming devices not only allows you, the user, to have more devices displayed, but it also improves our device detection models.