This document discusses AWS IoT Device Shadows and Rules Engine. It provides an overview of how shadows store and sync the reported and desired states of IoT devices. Rules Engine allows filtering and routing messages to other systems like S3, DynamoDB, Elasticsearch. The document includes examples of using shadows and rules to analyze data from a wind farm application. It highlights that shadows enable setting device state when offline and rules help store and integrate large IoT data volumes with big data services.
9. AWS IoT Device Shadow - Simple Yet Powerful
{
"state" : {
“desired" : {
"lights": { "color": "RED" },
"engine" : "ON"
},
"reported" : {
"lights" : { "color": "GREEN" },
"engine" : "ON"
},
"delta" : {
"lights" : { "color": "RED" }
} },
"version" : 10
}
Device
Report its current state to one or multiple shadows
Retrieve its desired state from shadow
Mobile App
Set the desired state of a device
Get the last reported state of the device
Delete the shadow
Shadow
Shadow reports delta, desired and reported
states along with metadata and version
10. 1. Device Publishes Current State
2. Persist JSON Data Store
3. App requests device’s current state
4. App requests change the state
5. Device Shadow sync’s
updated state
6. Device Publishes Current State 7. Device Shadow confirms state change
AWS IoT Device Shadow Flow
15. Rules engine: Extracting value from messages
• Filter messages with certain criteria
• Transform the payload of messages
• React based on messages
• Move messages to other topics
• Move messages to other systems
• Predict changes based on trends
16. Rules engine: Move messages to other systems
Invoke a Lambda function
Put object in an S3 bucket
Insert, update a
DynamoDB table
Publish to an SNS topic
or endpoint
Publish to an Amazon Kinesis
stream (and to EMR and Spark)
Publish to Firehose
Republish to AWS IoT
Publish to Amazon ES
Capture a CloudWatch
metric or change an alarm
Write to SQS queue
17. Rules engine: Move messages to other systems
Publish to Amazon Elasticsearch
Service
Write to SQS queue
Publish to an SNS topic
or endpoint
18. A rule in AWS IoT
aws iot create-topic-rule --rule-name WindFarmData --topic-rule-payload
{
"sql": "SELECT datapoint_id, current, rotation_speed, location,
topic(2) as turbine_id, timestamp() as time,
FROM 'wind-farm/+/data'",
"description": "Save data from wind turbines into Elasticsearch",
"actions": [
{
"elasticsearch": {
"roleArn": "arn:aws:iam::123456789012:role/aws_iot_windfarm",
"endpoint": "https://my-es-endpoint.us-west-2.es.amazonaws.com",
"index": "turbine-data",
"type": "measurement",
"id": "${CONCAT(datapoint_id, '_', turbine_id)}"
}}]}
RouteFilter, transform
26. Takeaways: Shadows
• Devices report state
• Applications set desired state
• Call APIs for devices, even when they’re offline
• Sync state with PubSub
• Consume deltas or complete state
• Use version numbers
27. Takeaways: Rules
• Use rules to store huge data volumes
• Rule actions integrate with big data storage services
• Use rules as triggers to drive your application
• Transform data moving from or to devices
• Supplement data with other data, predictions
Throughout this talk, we’re going to stick to the classic IoT story of connected wind farms.
Here we can collect data about the power output of the turbines around the world, and forecast, plan, and react to changes and supplement with power from other sources. The data coming off of wind turbines can drive predictions around when maintenance is needed using machine learning, which will talk about more later.
We’ll be coming back to this demo as we cover the different topics in IoT, but here’s part of it end-to-end so you can see what we’re talking about.
Show wind turbine, turn on the fan to power it. Show a webpage showing power produced, RPMs over time.
Also show Kibana elastic search with where it is on the world, and how much power is being produced in that part of the world
https://s3-us-west-2.amazonaws.com/aws-iot-wind-farm/display.html
https://s3-us-west-2.amazonaws.com/aws-iot-wind-farm/display.html
As a quick aside, I wanted to call out one interesting “Easter Egg” in the slide. I’ve filled in “Fowler, IN” as the power grid attribute because that’s where one of AWS wind farms lives. AWS has a couple more coming online in 2016 and 2017, and a solar farm as well. When online, they’re expected to generate 1.6 MWh of renewable energy annually into electric grids that power current and future AWS cloud data centers. If you’d like to learn more about them, check out the AWS Sustainability page.
As you can see, several pieces in IoT that work together, we’ll talk about how to use each of them in practice in a bit.
But what you get out of this is a fully managed, serverless, automatically scaling, pay as you go.
Okay, so sending commands seems pretty non-interesting. But remember those constraints at the beginning, around connectivity?
A shadow lives in the cloud for your applications to interact with in real-time, and when the device comes back online it sees what you tried to do with it while it was offline.
Desired, reported, delta
Versions – ordering, optimistic locking
Metadata – things like timestamps on last updates that can be used for staleness of the update
Why shadows
Query device state: Devices aren’t always on, are sometimes slow to reach (query)
Change device state: Devices might not be on to get the update right now.
Desired and reported state concepts
The basic API for setting and getting, topics for pushing and updating
Demo: Temperature control
It turns out this concept works well in practice, too. iRobot uses the shadow for all of their asynchronous communication with the robot. You can imagine things like changes to the vacuum schedule state as being a pretty natural fit for the shadow. Users want to be able to update settings even if the robot is offline, but when it comes online, it should pick up those changes. And if the robot is online, it will get the update right away.
Firehose: put your data into Redshift, a hosted data warehouse service,
Kinesis: real-time streaming analytics
Elasticsearch: we saw that in action for real-time full-text search and analytics
DynamoDB: a NoSQL datastore for low-latency, extremely scalable indexing.
CloudWatch metrics: create your own custom graphs and metrics dashboards. IoT emits some metrics there for you already, but this way you can emit your application monitoring metrics along with it.
Republish: even looser coupling between devices. Maybe devices only know how to interact with their own shadow or their own topic. Rules let you glue together devices that weren’t designed to talk to each other in the first place.
Lambda: anything else. Whatever code you want to write, but serverless!
INTEGRATION (their existing SQS and Elasticsearch clusters)
For example, Rachio is a company that makes smart connected sprinklers. I have one, and it works great for doing things like skipping watering when there’s been enough rain and humidity lately. Anyway, Rachio found AWS IoT to be great as a replacement for managing their own MQTT broker, since they weren’t getting the availability and scalability they needed before. Among other things, they use the rules engine to route millions of Rachio Smart Sprinkler Controller messages every day into SQS for processing by their back-end.
They also use Elasticsearch for event data logging and auditing. Today their SQS workers handle putting their data into their own Elasticsearch clusters, since they had this in place before migrating into AWS IoT, but if you’re getting started on the AWS IoT platform, you can configure a rule to put the data straight into Elasticsearch – just like the connected wind farm you saw earlier.
we just blogged about this, but we recently added elastic search as a native action in the rules engine.
Before this, customers had to daisy chain a set of integration rules (kinesis -> lambda -> ES)…
What’s this get you? Full text search across all your sensor data, with aggregate queries
We’re introducing a new function in the rules engine… the predict function.
AML has something called a real time prediction endpoint (last trained model)...
Rules Engine can make a callout to AML and use that value to determine if the sensor message meets it’s criteria
Example... Temp > 70 and predicted maintenance window < 30 cycles...
Actions... Lambda, SNS mobile push, etc