How Sense Learns About Your Devices

By collecting 4 million data points every second, Sense builds up a very detailed library of device signatures over time that let it recognize which devices are on in the home and when. Many of you have asked us if you can help speed up this process by manually identifying devices. Here, we’ll explain the data science behind our approach, and how you can help Sense learn and contribute to the accuracy and speed of our device identification algorithms.

Context provides important clues

In order to accurately identify a device, Sense needs to see repeated occurrences of the device in its typical operating context. From the standpoint of an electrical signature, many devices are not simply on or off — rather, they operate in a range of different states. Your typical dishwasher could be on, and the motor just kicked on; or it could be in rinse, drain, or standby mode. Each of these states looks very different to Sense.

Seeing a device’s full cycle and varied states helps our data science team identify it. Sense also uses its knowledge of other identified devices to determine which one is which. For example, your dryer is likely to run after your washer. Sense looks for these sorts of contextual time patterns to help identify your devices. The real-world operation of some devices contains useful clues that may be absent during an artificial training process, like enabling a "training mode" and then turning an appliance on and off multiple times in a row. For example, microwave ovens usually are activated for multiples of 30 seconds, but a purposeful “training session” would probably run the oven for much shorter bursts in consecutive order. That would remove the important contextual cues that can help make device detection more powerful.

More time = more data = better detection

Sense needs to see many occurrences of a variety of your home appliances turning on and off before it begins to distinguish them as unique items. This is because Sense is learning about your devices in the context of your particular home. Sense uses unsupervised machine learning to detect your devices. The qualifier “unsupervised” refers to the fact that there is no “ground truth”; there are no alternative methods to definitively determine each device that is on or off in your home. Sure, you might be able to tell Sense with certainty when your toaster is on, but doing this for every single device, each possible state for every device, and every possible combination of devices that might be on and off within your home at a given time, would be quite an undertaking! However, leveraging this advanced machine learning technology to allow the algorithms to "teach themselves" opens up an incredibly bright future for this technology.

Sense is like cat videos. No really.

To provide a related example, scientists at Google X trained an “artificial brain” to identify cat videos on YouTube by feeding it over 10 million videos. Instead of conducting pattern matching against videos labeled with specific features such as “human”, “cat,” or “cars,” — referred to as the training set — the system classified the images based on the patterns it found in the data alone. Again, it is important to note that Google X did this by using the images in their natural context. The process would simply not work if, say, a set of cat drawings had been used for that goal. To get a feeling for the concept of unsupervised learning, imagine you’re a little kid helping your family with the laundry and you’re surrounded by an assortment of socks. You don’t need to know the names of colors to begin to group blue socks with blue, white socks with white, and so on. Although device detection is considerably more difficult than sorting laundry, thankfully we don’t need quite as much data as Google in order to train Sense!

When you consider the number of household devices, the many states of each device, and the variation between devices of a single type; it starts to become clear why Sense takes time to learn.

How can I help Sense learn?

We have introduced a variety of features that take advantage of user input to help improve device detection in your home and for the entire userbase.

• Network Identification allows Sense to see some of the simple “handshake” messages put out by your networked devices.
• Integrations with smart bulbs from Philips Hue and smart plugs from TP-Link Kasa and Belkin Wemo will net you instant detections for connected devices and provides great data to the Data Science team.
• Renaming your devices, supplying the make/model, taking advantage of the Community Names feature, and filling out your Home Details feeds the Data Science team great data that improves detection for everybody.
• When Sense finds a device, but you’re finding the detections to be inaccurate, you can report it as “not on.” This feeds our Data Science team valuable information so they can continue to refine the detection model.

Remember, even without native detections, you can still take advantage of Sense insights. The Power Meter is a fantastic tool that provides a real-time view of your energy consumption. Try turning on and off your devices while watching in the Power Meter, to identify how much they consume. You can do the same for the “Always On” devices in your home, identifying how much they’re costing you every day.