Building Resources 101

Background

Our VP of engineering, Jacob Leverich, wired his house with IoT devices: temperature, contact, and motion sensors. All of the data from those devices are fed into Observe. In this brief walkthrough I’m going to show you how that data gets shaped and related together. While Observe is not an IoT product, this is a fun and relatable way to show how things work.

Video Walkthrough TL;DR

Getting the data in

There is a small AWS Lambda function that scrapes data from the Samsung SmartThings API. This function POSTs JSON events to the Observe HTTP event collector. Every time one of the devices reports a measurement it looks something like this:

Typical of most measurement data, these events lack context. Humans don’t think in terms of “deviceId": "fb56027d-7517-4855-bdc3-e12326ce99d1“, they want to know that their living room is too hot. To solve this problem, we will also periodically ingest metadata from the SmartThings API. This is important so that we can handle device changes as well as new ones that might be spun up. For example, here is an event that tells us what device is represented by deviceId:

Shaping Data

Now that we’ve explored the way that these events are structured, let’s shape them into things that we actually care about. We will start by looking at the deviceInfo events. They contain information about each device. For example, the name, manufacturer, capabilities, and sensor type.

Let’s filter down to the device info events by selecting the JSON value “DEVICE_INFO” in the console and choosing “Show only this value” from the menu.

Now that we’ve filtered down to these events let’s extract some information from them. I’m going to select the label, locationid, roomid, and device type.

Now we have something that is easier for a person to look at. Instead of staring at a ball of JSON we have structured the data into a table. Still, we need to go a bit further. Let’s turn these events into a Resource. A resource is a “thing” that we care about. It is our way of tracking states and relationships over time. This could be an IoT device, container, user, build, session, really anything that changes over time.

I’m going to select the deviceID as the primary identifier and add the label, type, and identifiers as fields. I’m also going to tell Observe to show the label of the resource instead of the identifier.

Instead of that list of events we had before, we now have an inventory of all of the IoT Devices. As changes are made, or new IoT devices are brought online, they will automatically be added to this list. Let’s go ahead and publish this Resource. I’m going to give it the name IoT Devices and click the publish button. Now it’s available for anyone else to use on this instance.

To demonstrate this, let’s take a look at all of the Resources in the system. We can now see that “IoT Devices” is listed, and if we click on its card we are taken to a dashboard that has automatically been generated for it.

Let’s return to the raw events so that we can shape the measurement data. Just like before I’m going to filter using the JSON console, only this time I’m going to filter down to the device events.

From these events I want to extract the deviceID, locationID, attribute, and value. Now we have a table that clearly shows each measurement type and value.

But these DeviceIDs are very opaque. Because we saved the devices as a resource, we can look up the name of each device. Let’s do that by selecting “link to resource” from the menu. Now we have names that are easy for a human to parse, like Bedroom and Garage door.

We can visualize this data too. I’m going to filter down to the temperature measurements and plot them by the device. Looking at data from the last 7 days shows off a nice diurnal pattern.

Like before, I can save this out as a dataset for other people in the instance to take advantage of. I’m going to name it “IoT Temperature” and publish it.

I hope that you found this brief example useful. If you’d like to learn more about Observe, and how it can get you on the path to better Observability, request trial access here.