Monday, March 18, 2019

Sensor-Ways Open Source IoT Devices

In this post we will introduce the IoT devices that can work and communicate smoothly with the open source IoT backend application "Sensor-Ways".
We will start with one useful device with both Gas and Temp monitoring capabilities.

Hardware Components:
- Arduino Wemos d1 or d1 mini or similar ESP8266 which contains the embedded WiFi chip.
- Gas sensor like MQ-2
- Temperature Sensor DHT11
- Buzzer (for local alarm)
- 5V Relay modules (1-2 optional in case we need to control other electric devices upon any action from the sensor reading or from the Sensor-Ways backend)
- Connection wires.
- Adapter 9V would be good to power the Wemos d1 board, optionally you can power it using USB connection or 9V Battery.



Connectivity:
1. MQ-2 or any to 5v, GND and A0  
2. DHT sensor to 5v, GND and D6  
3. Optional Buzzer: GND and D7
4. Optional Relay 1: GND & D5 
5. Optional Relay 2: GND & D8 or D7 


Device Code:
You will need the 2 files in the same folder:


Code Compile & Upload:
Using Arduino IDE compile the code and ensure everything is working fine. Then upload the code to a connected board using USB.
Note: You may need to install some of the libraries as needed in the import section in the code.

Device Configurations:
The default code is configured to connect 2 sensors; Gas & Temp and no relay modules are connected, you can change this easily by the 2 variables:
//Connected Sensors Count 
int connectedSensors = 2;
//Connected Relay Modules Count 
int connectedDevices = 0;

You can also update the other settings like WiFI SSID, password, Sensor-Ways backend deployment IP/URL and Port and Device Id & Password.
Note: In the previous post we have deployed our Sensor-Ways application, so you can easily get the IP address and HTTP/HTTPS port.
Also we have demonstrated the steps required to register a new IoT device where we get at the end device Id and device password, all you need to do is to follow these steps again in the backend IoT Sensor-Ways server:
  1. Create a new Device Model with the same exact features supported in our IoT device (Gas and Temperature Sensors without any control device attached).
  2. Add new unique bar code attached/mapped to this device model (once for each IoT device)
  3. Go to Manage Devices and Add new device using this bar code (it will auto detect the associated device model)
  4. Once the device is registered, you will need to use the device id and password to update your IoT device to connect to the system
Here is sample of defining Gas-Temp device model:


- The other option to configure the device during runtime by deploying the code to the IoT device and it will allow you at 1st time to configure the required values at the 1st run; device Id, password, WIFI SSID and WIFI password.
As simple as: Open the serial monitor in the Arduino IDE, send commands like: id=value, pass=value, wifi=vale, password=value, then send restart=true (1st time you will need to do a manual restart for the device after the code uploaded otherwise you will get an error)
You can write help=true to show all the possible commands.

This serial connection is very useful when you ship your IoT devices to the customers with pre-configuration to your IoT server, all you need during installation is to connect the device and configure the required customer variables such as device and WiFi identifiers.

Note: For production you will need to switch the debug messages by commenting one line in the variables section that define the debug:
#define DEBUG
To looks like:
//define DEBUG

Monitoring The Device & Device Management:
Now if we go to Sensor-Ways backend application and open Manage Devices we can see the device status and last message received from the device and last ping (last seen).


If we click on that device we can see more information in the device details page and we can also send some commands or do some sort of device management.



When you have enough messages, the graph icon will appear where you can see the pattern in your device messages.

You may try to send some commands such as request new update message from the device or restart the device, you can also edit the thresholds for alert for different sensor (per device).
You will get notifications upon any alert (values that violate the thresholds) also when the device in online or offline or sending error data, etc.. you can control all these messages from the preferences:



The platform has a lot of other features like defining workflow based on the sensor input values, or schedulers,  also building dashboards, simulators, etc.. you may need to spend reasonable time to fully understand the different features.

In subsequent posts, we will demonstrate many other IoT devices that enrich our IoT platform capabilities.


Saturday, March 9, 2019

Sensor-Ways Open Source IoT Platform

Sensor-Ways is my new open source project, created to provide end-to-end IoT capabilities and enable you to create different IoT applications.

This is an example of the devices list page.


In this post we will list the different capabilities in this platform which uses MySQL DB and Glassfish server for the runtime environment, all open source based technologies.
The platform aim to use Arduino-based chips which can be customized as per our application needs, so the whole system enables the customization and upgrade ability of both hardware and software components of the IoT device.

List of features:

- Multi-tenant platform allow you to manage different and separate accounts/customers.
- Account/User Management including activating/deactivating users, assign either read-only/read-write to different users.
- Ability to define Device Models with different sensors and control units.
- Ability to define bar codes for each device model to enable the on-boarding of end-user devices.
- Ability to manage IoT Devices statuses, send command, request messages, etc..


- Complete Device Management capabilities including: Control IoT Devices, Send/Recieve messages/commands, and enabled OTA firmware upgrade.
- Define thresholds for alerts per each IoT device.
- Define smart rules to be executed locally inside the device based on the sensors' readings.
- Built-in notification alerts


- Using gmail for sending email notifications
- Build workflows across different devices.
- Build schedulers to execute actions on different devices.
- Build device groups as a virtual grouping to control devices as one unit.
- Build customized dashboard for each user with nice and simple graphs.


- Assign a notification user per each device which gives high flexibility.
- Build different simulations to simulate the IoT devices to build the IoT applications.
- Upload new firmware for any model and define the required actions upon this new firmware upload.
- Build customized reports as per needed.
- Navigate and filter different audit records.
- Full control over system jobs and their configurations e.g. offline detection of devices, purge old records, scheduler execution, etc.
- Support different languages for the GUI and Notification templates.
Communication Protocol
The protocol is documented in the application, in protocol.jsp page where you can clearly understand the different parameters, header values and responses.
4 Main message types: login, update message, ping message and update firmware message.

Project Repository

The project is hosted in GitHub public repository:
https://github.com/osa-ora/sensor-ways

Installation Guide: Development Environment

- The following steps for development environment setup, as this project is Java based project, and it is built using NetBeans 8.2 IDE.

1- Install MySql DB 5.7 (if later version, you will need to review connection string, driver and fix it for proper DB connectivity)
2- Create the required schema "iot" for the application and grant the application user access to the database "iot_user"
  •       Create "iot_user"
  •       Create iot schema and run the schema creation script file "schema_tables.sql"
  •       Populate the iot schema for LOV data by running the script file "lov_data.sql"
  •       Grant the user "iot_user" full privileges to the iot schema
3- Create default objects so the application can run including: tenant_settings, users and system_config tables.
Here is the sample entries for these tables:

INSERT INTO `tenant_settings` VALUES (1,100,'Development Space', 1,150,15,20,'Africa/Cairo',0,'2019-01-31 16:21:06',5,20,1,0,0,0,0,5,0,0,0,'YOUR_EMAIL',15,1,1)

//Replace the email with your email.

INSERT INTO `users` VALUES (1,'Osama Oransa', '$31$16$nnmJyLDHWYGSGUF7gyrBjc2aw4JzkoB-PL5QCn_nKtA' ,'YOUR_EMAIL',100,1,0,'2019-02-03 11:40:26',1,'2018-06-01 18:07:43','2018-11-21 20:03:39' ,1,null,1,1,1,1,1,1,1,2,'0:0:0:0:0:0:0:1')

//Replace "Osama Oransa" and email with your data, email should match the email that you used in the tenant setting.

The default password is 123, no need to change it, because you can change it once the GUI is ready.

INSERT INTO `system_config` VALUES (1,'1.0','1.0',1,'smtp.gmail.com','587',1,'GMAIL_EMAIL','GMAIL_PASSWORD',2,'2018-11-10 15:47:46','1.0','SERVER_IP',SERVER_HTTPS,SERVER_HTTP,'Africa/Cairo');

//These settings is mainly for sending emails and for server information, you need to populate them all or disable the email sending.
//Server information could be using the local or public IP Address of the target server of this deployment, this is very critical information for OTA device firmware upgrade.
NOTE: Gmail account for sending emails must be security enabled to allow external applications to send emails.

4- Import the project/open it using NetBeans IDE8.x or later.
5- Change the Glassfish resource file "glassfish-resources.xml" to point to the correct MySql DB and to use the proper "iot_user" credentials.
6- Build & Deploy the application
7- Login using your email, password=123 and development-identity=100


8- Once logged-in change your password, and start to use the system.

Installation Guide: Production Environment

- Ensure DB setup is correct and in place.
- You need to export the WAR file and deploy it from the Glassfish Admin GUI.



Steps To Add an IoT Device:

From the Server side we need to do the following steps:

  1. Create a new Device Model with the same exact features supported in your IoT device (once per device model).
  2. Add new unique bar code for this device model (once for each IoT device)
  3. Go to Manage Devices and Add new device using this bar code (it will auto detect the associated device model)
  4. Once the device is registered, you will need to use the device id and password to update your IoT device to connect to the system.


We will see how to use the device id and password plus other important attributes in the next post where we will discuss the Arduino-based IoT Device Setup.

Features still in development:

The following areas are still in development:
- Add support to MQTT/JMS and other required protocol (Add different end points for each protocol)
- Expose REST APIs for all operation (for management + mobile clients)
- Build Mobile Clients that consumes the REST APIs
- Add push notification as one of the notification channels
- Add batch processing to some modules such as device registration.
- Implement IoT device change password from the server
- Add GUI language switch e.g. Arabic/English
- Build some default applications
- Enhance or build a new GUI
- Improve dashboard and graph features by using ready open source modules.
- Add better reporting capabilities.
- Add billing reporting capabilities (to produce consumption reports)
- Add some validation in the front-end, and backend.
- Fix the workflows in the level of device group