This repository will not be updated. The repository will be kept available in read-only mode.
In this code pattern, we will setup and create a web application to visualize IoT device data and view trends and stats of device fields across days. The IoT industries are looking for ways to analyze the use of IoT devices, and would like better understand the usage of devices. This code pattern will demonstrate using IBM solutions to read and store IoT device data, and then build an application on top of it. The code pattern uses services offered on IBM Cloud such as IBM Watson IoT Platform and Cloudant DB, in addition to deploying the application to the IBM Cloud. Once setup, the application displays the time-series field data as plots, showing device data trends and statistical analysis.
First, we will create an IBM Watson IoT Platform service which provides a platform to manage IoT devices and the data being sent across those devices. This code pattern provides directions on creating dummy IoT devices in the Watson IoT Platform, and then simulating data for those devices using a simulation feature in the IoT platform. We are interested in certain fields as part of payload for the devices to come through, as the application is designed to read those fields. Alternatively, we have provided a sample data set so you can quickly build and test out the application.
Next, we will store the devices' fields data from the Watson IoT platform into Cloudant DB. The Cloudant DB is a NoSQL JSON document store that is optimized for handling heavy workloads of read and write in the cloud. In this code pattern, we will walk through the process of creating a Cloudant DB service on the IBM Cloud, and retrieve credentials to access it through applications. The Cloudant DB interface can be launched through the service, which allows to directly manage and view our data. Once our Cloudant DB is created, we will go through steps on configuring Watson IoT Platform to store the IoT device data into it.
Once we have completed the setup, with our data coming through the Watson IoT Platform and into the Cloudant DB, we are ready to run the application to view the visualizations of data. The application first requests the user to create a dataset based on the data now being stored in the Cloudant DB. This can be viewed as our first layer of filtering assuming the IoT data scientist/analyst would like to focus on particular devices and dates. The device data could get large so it can be divided into different datasets for more focused analysis.
After creating a dataset, the user can use this application to create different visualizations for the data. This includes viewing the raw data for a device's data value across time, and then creating hourly trends plot that capture the behavior of the data per hour. The hourly trends looking at the max, min and moving average of the data which allows to capture the general trend of the field across time and capture any anomalies. Other analytical plots include comparing device's field stats (min, median, max) across days and viewing correlation between fields for a device across days. The Plotly.js library is used to create these visualizations of the data based on user inputs. The Plotly.js provides a great way to display data visually through numerous plot types and ability to manage and enhance on plots.
This web application is built on Python Flask framework, with Javascript(JS) and HTML frontend. The Python backend allows to create libraries (plotdata.py and dataset.py), which parse and analyze the JSON data from the Cloudant database. The frontend application provides a separate page for each plot type, and an associated JS script for each page in the scripts
folder under static
(i.e deviceCorrelationAnalysis.js). The user inputs are sent through an Ajax call to the Python backend (run.py) which retrieves the respective data and returns it. This includes the data to be plotted, which is captured by the JS script for the page and shows how to plot using the Plotly.js library. These plots are then displayed through the HTML page for the plot.
This code pattern can be useful to developer's looking to enhance IoT analysis skill, for IoT data scientists/analysts looking to create their own customized application, and anyone with interest in creating a visual analytical application.
When the reader has completed this code pattern, they will understand how to:
- Setup IoT devices on IBM Watson IoT Platform and simulate device's data
- Create Cloudant service on IBM Cloud, and using as data store for IoT device data
- Develop an application to parse JSON data from Cloudant DB using Python
- Display Plotly.js plots through a web application
- The IoT device data is stored in Cloudant database from the IBM Watson IoT Platform
- The data from Cloudant database is used to create Plotly visualizatons
- The plot is displayed through the Web UI based on user requests
- The user can view the plots and perform analysis on each plot through the web UI
- IBM Watson IoT Platform: This service is the hub for IBM Watson IoT and lets you communicate with and consume data from connected devices and gateways. Use the built-in web console dashboards to monitor your IoT data and devices.
- Cloudant NoSQL DB: A fully managed data layer designed for modern web and mobile applications that leverages a flexible JSON schema.
- Plotly.js: A high-level, declarative charting library with 20 chart types, including statistical graphs, 3D charts and SVG maps
- Python Python is a programming language that lets you work more quickly and integrate your systems more effectively.
- JavaScript JavaScript is a prototype-based, dynamic language, most well-known as the scripting language for Web pages
Use the Deploy to IBM Cloud
button OR create the services and deploy the application locally.
You can deploy the application and create services directly on IBM Cloud using the 'Deploy to IBM Cloud' button. This is alternative to creating services individually and then deploying the application using cloud foundry.
- Create an IBM Cloud account and directly deploy the application using the button bellow. This will create the IBM Watson IoT Platform and Cloudant DB services for you and connect to your application.
- Here you can provide your application and toolchain a name. You can choose the
Region
,Organization
andSpace
where you would like to deploy the application. Then clickDeploy
.
- Next, you can view your toolchain. We can see the progress of deployment by clicking
Delivery Pipeline
.
- The delivery pipeline will show the status of each stage. You can view logs and history if any issues. Once all stages pass, then your application has been successfully deployed.
- In your IBM Cloud dashboard, you can now find your application. You can open the application which provides an overview of the application as hosted in IBM Cloud.
- You can view the services connected to this application by clicking
Connections
on the left menu.
Once your application and services are created in IBM Cloud, you will need to do the following to setup your IoT data in the Cloudant DB:
Or if you would like to use a sample dataset then proceed to this step:
Once you have completed these steps, you should be able to create dataset in your application to start viewing trends and analysis of the IoT device data.
NOTE: These steps are only needed when running locally instead of using the
Deploy to IBM Cloud
button.
Note Two approaches to setting up and running the application are below. Option 1
teaches you how to simulate devices and create a sample dataset manually. Option 2
acclerates the process of running the application by taking advantage of using an already made dataset that is included in the repo.
Steps (Option 1) for setting up IBM Watson IoT Platform with Cloudant DB, and running the application
- Create IBM Watson IoT Platform service on IBM Cloud
- Create and simulate devices on IoT Platform
- Create Cloudant DB on IBM Cloud
- Configure Cloudant DB as data store for IoT device data
- Run the web application
- About the application
- Deploy application to IBM Cloud
- Create Cloudant DB on IBM Cloud
- (Option 2) Populate Cloudant DB with sample dataset
- Run the web application
- About the application
- Deploy application to IBM Cloud
The IBM Watson IoT Platform service provides a dashboard to manage and configure devices, read data transmitted by the devices and numerous features to managing IoT devices. First, we would like to create the IBM Watson IoT Platform service in IBM Cloud.
- You can access IBM Cloud by going to https://console.bluemix.net/. If you do not have an IBM Cloud account you can create one.
- If you do have an IBM Cloud account or have created on, then login to create the service. You can search for the service in
catalog
and choosingInternet of things
. Here we would like to pick theInternet of Things Platform
- Or alternatively can go directly to the link here to create the service. Give your service a name and choose the
region
,organization
andspace
.
- You can pick the
Lite
plan for this code pattern to get started with Internet of Things Platform.
- Once your service is created, you can launch Watson IoT Platform by clicking the
Launch
button. Additionally, you can access documentation regarding the IoT Platform by going toDocs
button.
- Once your Watson IoT Platform, it should take you to a similar dashboard below. Now you are ready to create devices and simulate data for the application.
Once you have created your IBM Watson IoT Platform service, you are ready to setup devices to transmit data. In this section, we will create a dummy device type, then create devices using the device type and then simulate data for those devices using the simulator feature in the IBM Watson IoT Platform.
The application in the code pattern is designed to handle payload as the following json:
"deviceId": "d6a82126d",
"timestamp": "2018-04-04T11:36:31.046Z",
"data": {
"connections": 58,
"deviceCount": 78,
"activeClients": 68
}
The deviceId
and timestamp
are submitted automatically by the IoT Platform, however we would like to configure the devices to send data for connections
, deviceCount
and activeClients
. The application will plot and analyze these fields over time per device.
Before creating devices, we would need to create a device type. Device types are intended to be groups of devices which share common characteristics.
- To add a
device type
, first go to the IBM Watson IoT Platform options on the left side of your dashboard and chooseDevices
.
- Next, go to
Device Types
on your screen, and click theAdd Device Type
button on the top right.
- Here we will go through options to create the device type. Under the
Identity
tab, chooseDevice
for type, as this code pattern is focued on devices. Provide aName
and an optionalDescription
, and then clickNext
.
On the Device Information
, you can add attributes to the device type. These attributes are optional. Then click Next
and Done
. The device type should be registered and you should see a similar screen as below.
Now, we will add device to the device type we created.
- You can choose
Register Devices
under your device type. Or go toDevices
in the menu, and chooseAdd Device
in the top right corner.
- In the
Identity
tab, choose the device type and provide aDevice ID
. Then clickNext
.
- You can choose to provide
Device Information
andAdd to Groups
. You can go through the steps by clickingNext
. Under theSecurity
tab, provide anAuthentication Token
. ClickNext
and thenDone
.
- Now your device is added to your IoT Platform.
Now we would like to setup our devices to transmit data. Ideally you would to like read in real data or can have scripts to simulate the data (see Additional Resources
at the bottom for more information). For this code pattern, we will go through steps to simulate data sent through by the devices, using simulator feature part of the Watson IoT Platform dashboard.
The IoT Platform provides a simulator feature to simulate the payload transmitted by the device. In this step, we will turn on the simulator and simulate numbers for the fields to be transmitted by the device.
- Go to the menu on the left side of your dashboard and select the
Settings
option:
- Here scroll down to the
Experimental Features
section. And toggle enable theActive Device Simulator
.
- Once you have enabled the simulation, you can update the payload and frequency of data sent by the device. Look for
Simulations running
tab at the bottom of screen in your IoT Dashboard. Bring that up to create the simulation, by selectingAdd First Simulation
.
First choose a Device Type
. Next, we will define the Payload
to include the fields for this code pattern: connections
, deviceCount
and activeClients
. The simulated data can look similar to below where it is generating random numbers for the fields, transmitting data every minute. You can provide an Event type name
. Once done, click Save
.
- Now for your device type, you can add devices for which you would like to run the simulation for. Click the
dropdown
arrow next to your device type, and chooseUse Registered Device
. Here you will see your devices and can choose the device you would like to simulate.
- To check your device simulation. You can click on your device under
Devices
, and then go to theRecent Events
tab. Here you can view the live data being transmitted for the device.
Now we are ready to store the IoT device data into a database. The IBM Cloud provide several database options including relational and non-relational options. For our data with the json payload, we would like to choose a NoSQL database such as Cloudant DB. The Cloudant DB provides for heavy read/write for an application, easy retrieval of data through API calls and a great interface to manage data directly. In this section, we will create a Cloudant DB on the IBM Cloud.
- Go to to your IBM Cloud dashboard. You can go to
catalog
and underData & Analytics
, you can findCloudant NoSQL DB
service.
- Or you can go directly to the service here to create the Cloudant DB. Give your Cloudant DB a
Service name
, and then choose theregion
,organization
andspace
- You can pick from different plans, depending on the size of your data. For this code pattern, we can use the
Lite
plan to get started. ClickCreate
at the bottom left to create the service in your IBM Cloud.
- This has now created the service. Through this service, you can launch the Cloudant interface, retrieve credentials for the service and create connections to applications in your IBM Cloud.
- To get service credentials, go to the
Service credentials
option on the left. Here you can generate credentials for the service by clicking onNew credentials
. Go ahead and create credentials for the service as we will need those for our application.
Now, we will configure our IBM Watson IoT Platform to setup Cloudant DB as daily store for our device data. The IBM Watson IoT Platform provides a straight forward way to create daily buckets for our device data. Once we are configured, we will have a database for each day in our Cloudant DB with our device data.
- In the IoT Platform, go to the menu and choose
Extensions
.
- Here, under
Historical Data Storage
, chooseSetup
. Here you will see all your databases in your IBM Cloud. Find the Cloudant DB you created to store the device data.
- Next, set up your
Bucket Interval
forDay
and then clickDone
- Now your cloudant DB is set to receive data from Watson IoT Platform. In your Cloudant DB, you should view a database for each day now:
- You should view similar data in your Cloudant json as below:
To quickly view IoT device data across days, this repository contains sample datasets. In the sample_datasets folder we will see two datesets, for the month of January and March. These contain ten days of data with a json datafile for each day. These files are populated with sample data from IBM Watson IoT Platform for particular devices.
This data can be can be populated into your Cloudant DB through the script write_json_to_cloudant.py.
In a directory of your choice, first clone the repo:
git clone https://github.com/IBM/iot-device-trend-analysis
Next update the script with you Cloudant credentials and to the dataset you want to load
-
Edit the script to add your Cloudant credentials
#cloudant credentials serviceUsername = "" servicePassword = "" serviceURL = ""
-
Update the script to choose the dataset you want to load between
January
orMarch
-
Edit to choose appropriate dates. This should be among the dates for which the data is present in the dataset.
Now you can run the script. Navigate to the sample_datasets folder, and run the script.
cd sample_datasets/
python write_json_to_cloudant.py
This should populate your Cloudant with a database for each day emulating the IoT device data obtained from IBM Watson IoT Platform.
Once the setup is complete with data coming into our Cloudant DB, we can run the application to start viewing the device data values and look for trends. To run the application, we will clone the repo, add the Cloudant credentials and then run the application through terminal.
In a directory of your choice, if you have not, first clone the repo:
git clone https://github.com/IBM/iot-device-trend-analysis
Create a .env
file in the root directory of your clone of the project repository by copying the sample .env.example
file using the following command:
cp .env.example .env
You will need to update the .env
file with credentials for the Cloudant DB which you created.
The .env
file will look something like the following:
#Cloudant NoSQL database
CLOUDANT_USERNAME=
CLOUDANT_PASSWORD=
CLOUDANT_URL=
Now you are ready to run your application. Go into this project's root directory
- Run
pip install -r requirements.txt
to install the app's dependencies - Run
python run.py
- Access the running app in a browser at http://localhost:5000/
The application is designed to create dataset from our Cloudant DB which we would like to plot and analyze, and then analyze through plots based on user inputs. The application provides to view the raw data and view analysis on the data including trends and statistical plots. In this section, we will look at creating dataset, analyzing the data through plots and look at the code.
Before analyzing the data, you will need to define a dataset. This includes dates and device Ids from your Cloudant storage. You can do this through the app by going to Create Dataset
link on the main page. This pulls all the dates
and deviceIds
directly from the Cloudant database. You can make selection for these fields and give a name to your dataset.
Or you can manually edit the datasets.json
to fill in your database info with dates, deviceIDs, IoT database initial and name for the dataset. The file should look like below:
{
"currentDataset": "New1",
"datasets": [
{
"databaseName": "iotp_nmghmm_default_",
"dataset": "New1",
"dates": [
"2018-01-28",
"2018-01-29",
"2018-01-30"
],
"deviceIds": [
"830ab1575",
"bc1a6275a",
"1d0c388d0",
"51bdc89e4"
]
}
]
}
Once you have defined your dataset, you are ready to analyze your data through the different options present on the homepage. Each analysis will ask for device(s) and date(s) to generate the plot. Once your plot is generated, you can explore different Plotly interactive options on the top right which can allow to download the plot, change the plot type and other actions. The Hourly Stats and Trends
options will allow to view how the hourly max, min, and average are behaving across time, with trend analysis showing change in the values per hour.
This web application is built using Python Flask framework. The repo consists of Python functions to retrieve the JSON data from Cloudant DB, and Javascript frontend to use the Plotly.js library. Here we'll give a summary of the code files in the repo.
-
run.py
: This routes the pages on the web application and manages all theGET
andPOST
commands made using Ajax on the Javascript scripts. These calls will then call the respective functions inplotdata
ordataset
library to return the requested data to the web frontend. -
plotdata.py
: This library contains functions that pulls data from Cloudant DB through API call and parses the data according to the function and input. These functions will return a list of JSON objects, where each object will havetimeStamp
field and data fields that we are interested in plotting. This library includes functions such asDevice_data_across_days
which pull raw data per device id for a start and end date, andHourly_stats_trends
which will create a JSON to plot the hourly stats and trends
For example the JSON object for Device_data_across_days
will be as follow with the raw data:
{
"deviceCount": 85.0,
"timeStamp": "2018-01-16T10:35:41.635Z",
"connections": 43.0,
"deviceID": "19ca0a0b6",
"activeClients": 65.0
}
The Hourly_stats_trends
function will return JSON object with the max, min, average, and change in these value for every hour (i.e slope). In addition it provides a time stamp for that hour right at 30 mins as the middle point for that hour.
{
"sumActiveClients": 253.0,
"plotTimeStamp": "2018-01-16T01:30:00.000Z",
"maxActiveClients": 84.0,
"date": "2018-01-16",
"avgSlopeLastHour": -22.4,
"minActiveClients": 26.0,
"maxSlopeLastHour": -18.0,
"hour": "01",
"minSlopeLastHour": -9.0,
"countEntries": 5,
"avgActiveClients": 50.6,
"deviceID": "19ca0a0b6"
}
-
dataset.py
: This library provides functions to managedatasets.json
, which includes pulling dataset information, adding dataset and setting the active dataset. The functions updates and retrieve thedatasets.json
file accordingly. -
JS scripts (i.e static/scripts/deviceDataPerDay.js)
: There is a JS script for each page of this application. The javascript makesGET
calls to update the page with dropdown options, ensures valid user input for the page, and makes the Ajax call to get the JSON data to plot. Then it uses Plotly.js library to create a plot for that page.
To create Plotly.js plots, we will first create traces for the plot, defining our x
and y
axis, and type
and name
for the trace. The type
determines what type of plot we would like i.e scatter, bar, boxplot. Next, we define data
for the plots, as an array of these traces, and layout
with the title for plot.
//define traces
var activeClientsTrace = {
x: timeStampArray,
y: activeClientsArray,
type: "scatter",
name: "activeClients"
};
var deviceCountTrace = {
x: timeStampArray,
y: deviceCountArray,
type: "scatter",
name: "deviceCount",
visible: "legendonly"
};
var connectionsTrace = {
x: timeStampArray,
y: connectionsArray,
type: "scatter",
name: "connections",
visible: "legendonly"
};
var data = [activeClientsTrace, deviceCountTrace, connectionsTrace];
var layout = {
title: "Device " + id + " from " + startDate + " to " + endDate
};
With your data
and layout
defined, you are ready to call the Plotly
library to create the plot. The plotly_div1
would be an id
in your html
page.
//create plot
Plotly.newPlot('plotly_div1', data, layout)
.then(
function(gd) {
Plotly.toImage(gd, {
height: 500,
width: 500
})
});
html (i.e /templates/devicePerDay.html)
: The html code is provided for each page, which is primarily to retrieve user input and display plots. It provides the css and JS with the ids and class for enhancing display and capturing user inputs.
To deploy the application to IBM Cloud, you will need Cloud Foundary CLI installed, so you can use command-line to deploy the application. The configuration for the deployment will be in manifest.yml
file, which we'll update first.
Here we will update the manifest.yml
file in the folder and replace with a unique name for your application. The name that you specify determines the application's URL, such as your-application-name.mybluemix.net
. Additionally - update the service names so they match what you have in IBM Cloud. The relevant portion of the manifest.yml
file looks like the following:
applications:
- path: .
memory: 256M
instances: 1
domain: mybluemix.net
name: Iot-Analytics
host: iot-device-analytics
disk_quota: 1024M
buildpack: python_buildpack
services:
- cloudant
The memory of 256M will allow you to run the application with the Lite
plan. However as data gets larger with the retrieval, you may have to increase memory to 512M or 1024M.
To deploy the application to IBM Cloud, we will use Cloud Foundry CLI. First login to your cloud foundry account
cf login
Next, in the command line use cloud foundry command to push the application to IBM Cloud:
cf push
This will take a few minutes and then provide the status of deployment. You can then go to the url to view the application.
This code pattern can be extended in several ways:
- Use for real world IoT device data
- Create predictive analysis for the data
- Update plots as device data streams into database
- Demo on Youtube
- IBM Watson IoT - Python library
- Using IBM Watson Analytics to visualize data from Watson IoT Platform
- Using IBM Watson Studio for Iot analysis
- Create devices on IoT Platform
- Simulate device data
- Plotly.js reference
To troubleshoot your IBM Cloud application, use the logs. To see the logs, run:
cf logs <application-name> --recent
This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.