Sensor-Ways Building Different IoT Devices

In this post we will build different variation of IoT devices that can work and communicate with the open source IoT backend application "Sensor-Ways".

The previous post summarize everything about building, compiling, configuring and monitoring the IoT device, so in this post we will focus on the hardware part and code.

All these devices can be either: Sensor-based only, Control-Only devices or Mixed Sensor and Control devices, so we will simplify all the devices to be sensor only and we will show an example of how to have a control device.

To connect a control device connect the relay the switch something on/off, you can connect up-to 2 relay modules as per the current design, but you can change it to accommodate more. Same for sensors, currently the supported is max 2 sensors connected, but this also can be change.
Therefore the following hardware components are for all devices and we will only list the required sensor beside each device:

Hardware Components:
- Arduino Wemos d1 or d1 mini or similar ESP8266 which contains the embedded WiFi chip.
- Buzzer (for local alarm)
- 5V Relay modules (1-2 optional in case we need to control other electric devices upon any action from the sensor reading or from the Sensor-Ways backend)
- Connection wires.

- Adapter 9V would be good to power the Wemos d1 board, optionally you can power it using USB connection or 9V Battery.

The following are optional connected components:

Connectivity:
1. Optional Buzzer (if local alarm is required): GND and D7

2. Optional Relay 1 (if control device is required): GND & D5 

3. Optional Relay 2 (if a second control device is required): GND & D8 or D7 

1) Safety Devices

These devices are mainly about home and workplace safety, for example you can monitor the temperature in different location so you can have early detection of fire and better quality of locating the exact fire location.

Same for gas leakage either for home or for industrial uses.

- Gas-Temperature Monitoring Device

We already discussed this device in the previous post in details:

https://osama-oransa.blogspot.com/2019/03/sensor-ways-open-source-iot-devices.html
Sensors:

1. MQ-2 or any to 5v, GND and A0  

2. DHT sensor to 5v, GND and D6  

Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-GasTempDeviceOTA.ino and IoTDevice-Common.ino

- Gas Monitoring Device

Sensor:
- Gas sensor like MQ-2 or any MQ sensor, Connected to 5v, GND and A0.
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-GasControlDeviceOTA.ino and IoTDevice-Common.ino

- Temperature Monitoring Device

Sensor:
- DHT sensor to 5v, GND and D6
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-TempControlDeviceOTA.ino and IoTDevice-Common.ino

2) Security Devices

These devices mostly fall into securing some locations such as restricted access doors, or home security.

- Door/Window Open Monitoring Device

This can monitor a window or a door to know if it is open or closed and send alert upon change of the status.
Sensor:
- Door magnetic sensor (see picture), Connected to 5v, and D6.
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-DoorControlDeviceOTA.ino and IoTDevice-Common.ino

- Laser Monitoring Device

To be posted soon.

- Drawer Control Device

To be posted soon.

- Camera Monitoring Device

To be posted soon.

- Motion Sensor Device

To be posted soon.

3) Environmental Devices

These devices is more about monitoring the environment and weather conditions, or reduce the consumption of water or energy.

The more you reduce the consumption of resources, the more efficient you help preserving the environment.

- Soil Moisture Device

By Monitoring the soil moisture degree, you can switch on/off water irrigation and reduce the water consumption according to the actual needs which can varies as per the weather conditions.

Sensor:
- Soil Moisture sensor (see picture), Connected to 5v, GND and A0.
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-SoilControlDeviceOTA.ino and IoTDevice-Common.ino

- Rain Monitoring Device

The use case to alert or automate something about rain starting, like retail shops that have some products outdoors or in the ports where products need to be covered in that situation.

Sensor:
- Rain drops sensor (see picture), Connected to 5v, GND and A0.
Device Code:

Same code like the Soil Moisture device but here we will care more about detecting any wet condition.

- Pollution Monitoring Device (Air Quality Sensor)

To be posted soon.

- Garbage/Trash Filling Monitoring Device

By Monitoring the trash filling level using ultrasonic sensor, we can detect if the trash need urgent emptying or not.

Sensor:
- Ultrasonic sensor (see picture), Connected to D6, D7, GND and V5
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-TrashSensorDeviceOTA.ino and IoTDevice-Common.ino

4) Control Devices

- One/Two Control Device

Sensor:
- No Sensors, Only connect 1 or 2 relay modules as: Relay 1: GND & D5, Optionally Relay 2: GND & D8 or D7
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-ControlDeviceOTA.ino and IoTDevice-Common.ino

5) Medical Devices

- Pulse & ECG Device

The idea behind this device is to capture either the ECG and send to the IoT backend or simply (more convenient) to detect heart rate, disconnected leads and flat rate, and set the alert to bradycardia or tachycardia or flat rate.
 Sensor:
- AD8232 heart monitoring connected to 3.3V, GND, A0, D6 and D7
Leads must be placed over the left arm, right arm and right leg to record the 3 lead electricity.
D6 & D7 detect disconnected leads if both are high (1)
A0 contains the signal/electricity recorded which you can either capture over time and send to the backend server or do some processing using a threshold detection for heart beats and send the heart rate to the backend server.
Device Code:
The code is simple as reading from A0 the electricity and either send a series of a specific duration e.g. 3 seconds or do some advanced analysis to detect HR and send only HR to the IoT backend.
Should be carefully used and never use such devices as a medical device as medical devices must be very reliable and a lot of efforts should be done for noise filtration as well.

6) Others Devices

The use case here is to detect morning and automate some work, or detect night to switch off something, etc.

- Light Sensor Device

To be posted soon.

- Parking Sensor Device

By Monitoring the parking slot using ultrasonic sensor, we can detect if the slot is empty or filled with something i.e. car.

Sensor:
- Ultrasonic sensor (see picture), Connected to D6, D7, GND and V5
Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-ParkingSensorDeviceOTA.ino and IoTDevice-Common.ino

- Smart Phone as an IoT Device

To be posted soon.

To Build Any Other devices, You need to pick the device with the same run-time requirement, and just change the sensor and may be other connections and build it in no time!

Sensor-Ways Open Source IoT Devices

In this post we will introduce the IoT devices that can work and communicate smoothly with the open source IoT backend application "Sensor-Ways".
We will start with one useful device with both Gas and Temp monitoring capabilities.

Hardware Components:
- Arduino Wemos d1 or d1 mini or similar ESP8266 which contains the embedded WiFi chip.
- Gas sensor like MQ-2
- Temperature Sensor DHT11
- Buzzer (for local alarm)
- 5V Relay modules (1-2 optional in case we need to control other electric devices upon any action from the sensor reading or from the Sensor-Ways backend)
- Connection wires.

- Adapter 9V would be good to power the Wemos d1 board, optionally you can power it using USB connection or 9V Battery.

Connectivity:

1. MQ-2 or any to 5v, GND and A0  

2. DHT sensor to 5v, GND and D6  

3. Optional Buzzer: GND and D7

4. Optional Relay 1: GND & D5 

5. Optional Relay 2: GND & D8 or D7 

Device Code:

From the GitHub code: https://github.com/osa-ora/sensor-ways-iot-devices/tree/master

You will need the 2 files in the same folder:

IoTDevice-GasTempDeviceOTA.ino and IoTDevice-Common.ino

Code Compile & Upload:

Using Arduino IDE compile the code and ensure everything is working fine. Then upload the code to a connected board using USB.

Note: You may need to install some of the libraries as needed in the import section in the code.

Device Configurations:

The default code is configured to connect 2 sensors; Gas & Temp and no relay modules are connected, you can change this easily by the 2 variables:

//Connected Sensors Count 

int connectedSensors = 2;

//Connected Relay Modules Count 

int connectedDevices = 0;

You can also update the other settings like WiFI SSID, password, Sensor-Ways backend deployment IP/URL and Port and Device Id & Password.

Note: In the previous post we have deployed our Sensor-Ways application, so you can easily get the IP address and HTTP/HTTPS port.

Also we have demonstrated the steps required to register a new IoT device where we get at the end device Id and device password, all you need to do is to follow these steps again in the backend IoT Sensor-Ways server:

1. Create a new Device Model with the same exact features supported in our IoT device (Gas and Temperature Sensors without any control device attached).
2. Add new unique bar code attached/mapped to this device model (once for each IoT device)
3. Go to Manage Devices and Add new device using this bar code (it will auto detect the associated device model)
4. Once the device is registered, you will need to use the device id and password to update your IoT device to connect to the system

Here is sample of defining Gas-Temp device model:

- The other option to configure the device during runtime by deploying the code to the IoT device and it will allow you at 1st time to configure the required values at the 1st run; device Id, password, WIFI SSID and WIFI password.

As simple as: Open the serial monitor in the Arduino IDE, send commands like: id=value, pass=value, wifi=vale, password=value, then send restart=true (1st time you will need to do a manual restart for the device after the code uploaded otherwise you will get an error)

You can write help=true to show all the possible commands.

This serial connection is very useful when you ship your IoT devices to the customers with pre-configuration to your IoT server, all you need during installation is to connect the device and configure the required customer variables such as device and WiFi identifiers.

Note: For production you will need to switch the debug messages by commenting one line in the variables section that define the debug:

#define DEBUG

To looks like:

//define DEBUG

Monitoring The Device & Device Management:

Now if we go to Sensor-Ways backend application and open Manage Devices we can see the device status and last message received from the device and last ping (last seen).

If we click on that device we can see more information in the device details page and we can also send some commands or do some sort of device management.

When you have enough messages, the graph icon will appear where you can see the pattern in your device messages.

You may try to send some commands such as request new update message from the device or restart the device, you can also edit the thresholds for alert for different sensor (per device).

You will get notifications upon any alert (values that violate the thresholds) also when the device in online or offline or sending error data, etc.. you can control all these messages from the preferences:

The platform has a lot of other features like defining workflow based on the sensor input values, or schedulers,  also building dashboards, simulators, etc.. you may need to spend reasonable time to fully understand the different features.

In subsequent posts, we will demonstrate many other IoT devices that enrich our IoT platform capabilities.

Sensor-Ways Open Source IoT Platform

Sensor-Ways is my new open source project, created to provide end-to-end IoT capabilities and enable you to create different IoT applications.

This is an example of the devices list page.

In this post we will list the different capabilities in this platform which uses MySQL DB and Glassfish server for the runtime environment, all open source based technologies.
The platform aim to use Arduino-based chips which can be customized as per our application needs, so the whole system enables the customization and upgrade ability of both hardware and software components of the IoT device.

List of features:

- Multi-tenant platform allow you to manage different and separate accounts/customers.
- Account/User Management including activating/deactivating users, assign either read-only/read-write to different users.
- Ability to define Device Models with different sensors and control units.
- Ability to define bar codes for each device model to enable the on-boarding of end-user devices.
- Ability to manage IoT Devices statuses, send command, request messages, etc..

- Complete Device Management capabilities including: Control IoT Devices, Send/Recieve messages/commands, and enabled OTA firmware upgrade.
- Define thresholds for alerts per each IoT device.
- Define smart rules to be executed locally inside the device based on the sensors' readings.
- Built-in notification alerts

- Using gmail for sending email notifications
- Build workflows across different devices.
- Build schedulers to execute actions on different devices.
- Build device groups as a virtual grouping to control devices as one unit.
- Build customized dashboard for each user with nice and simple graphs.

- Assign a notification user per each device which gives high flexibility.
- Build different simulations to simulate the IoT devices to build the IoT applications.
- Upload new firmware for any model and define the required actions upon this new firmware upload.
- Build customized reports as per needed.
- Navigate and filter different audit records.
- Full control over system jobs and their configurations e.g. offline detection of devices, purge old records, scheduler execution, etc.
- Support different languages for the GUI and Notification templates.
Communication Protocol
The protocol is documented in the application, in protocol.jsp page where you can clearly understand the different parameters, header values and responses.
4 Main message types: login, update message, ping message and update firmware message.

Project Repository

The project is hosted in GitHub public repository:
https://github.com/osa-ora/sensor-ways

Installation Guide: Development Environment

- The following steps for development environment setup, as this project is Java based project, and it is built using NetBeans 8.2 IDE.

1- Install MySql DB 5.7 (if later version, you will need to review connection string, driver and fix it for proper DB connectivity)

2- Create the required schema "iot" for the application and grant the application user access to the database "iot_user"

•       Create "iot_user"
•       Create iot schema and run the schema creation script file "schema_tables.sql"
•       Populate the iot schema for LOV data by running the script file "lov_data.sql"
•       Grant the user "iot_user" full privileges to the iot schema

3- Create default objects so the application can run including: tenant_settings, users and system_config tables.

Here is the sample entries for these tables:

INSERT INTO `tenant_settings` VALUES (1,100,'Development Space', 1,150,15,20,'Africa/Cairo',0,'2019-01-31 16:21:06',5,20,1,0,0,0,0,5,0,0,0,'YOUR_EMAIL',15,1,1)

//Replace the email with your email.

INSERT INTO `users` VALUES (1,'Osama Oransa', '$31$16$nnmJyLDHWYGSGUF7gyrBjc2aw4JzkoB-PL5QCn_nKtA' ,'YOUR_EMAIL',100,1,0,'2019-02-03 11:40:26',1,'2018-06-01 18:07:43','2018-11-21 20:03:39' ,1,null,1,1,1,1,1,1,1,2,'0:0:0:0:0:0:0:1')

//Replace "Osama Oransa" and email with your data, email should match the email that you used in the tenant setting.

The default password is 123, no need to change it, because you can change it once the GUI is ready.

INSERT INTO `system_config` VALUES (1,'1.0','1.0',1,'smtp.gmail.com','587',1,'GMAIL_EMAIL','GMAIL_PASSWORD',2,'2018-11-10 15:47:46','1.0','SERVER_IP',SERVER_HTTPS,SERVER_HTTP,'Africa/Cairo');

//These settings is mainly for sending emails and for server information, you need to populate them all or disable the email sending.

//Server information could be using the local or public IP Address of the target server of this deployment, this is very critical information for OTA device firmware upgrade.
NOTE: Gmail account for sending emails must be security enabled to allow external applications to send emails.

4- Import the project/open it using NetBeans IDE8.x or later.

5- Change the Glassfish resource file "glassfish-resources.xml" to point to the correct MySql DB and to use the proper "iot_user" credentials.

6- Build & Deploy the application

7- Login using your email, password=123 and development-identity=100

8- Once logged-in change your password, and start to use the system.

Installation Guide: Production Environment

- Ensure DB setup is correct and in place.
- You need to export the WAR file and deploy it from the Glassfish Admin GUI.

Steps To Add an IoT Device:

From the Server side we need to do the following steps:

1. Create a new Device Model with the same exact features supported in your IoT device (once per device model).
2. Add new unique bar code for this device model (once for each IoT device)
3. Go to Manage Devices and Add new device using this bar code (it will auto detect the associated device model)
4. Once the device is registered, you will need to use the device id and password to update your IoT device to connect to the system.

We will see how to use the device id and password plus other important attributes in the next post where we will discuss the Arduino-based IoT Device Setup.

Features still in development:

The following areas are still in development:
- Add support to MQTT/JMS and other required protocol (Add different end points for each protocol)
- Expose REST APIs for all operation (for management + mobile clients)
- Build Mobile Clients that consumes the REST APIs
- Add push notification as one of the notification channels
- Add batch processing to some modules such as device registration.
- Implement IoT device change password from the server
- Add GUI language switch e.g. Arabic/English
- Build some default applications
- Enhance or build a new GUI
- Improve dashboard and graph features by using ready open source modules.
- Add better reporting capabilities.
- Add billing reporting capabilities (to produce consumption reports)
- Add some validation in the front-end, and backend.
- Fix the workflows in the level of device group

Building Your Own Smart Home Solution

In this series of blog posts, we will explain how you can build your own Smart Home Solution with almost low cost.

High Level Architecture
The High Level Architecture is composed of the following:
- Arduino devices (IoT devices) that collect sensor data and perform actions.
- Back-end application : cloud hosted in a free tier and will be able to view messages and send commands to your IoT devices, we will build our application using JavaEE so it can be easily hosted in any cloud provider.
- Mobile Application : another way to communicate with your devices, it is a client of the back-end application, we will build sample Android App as an example, you can build another iOS application to use according to your needs.

Let's start by identify our platform requirements:
Hardware components
- Arduino board + ESP8266 shield for WiFi connectivity: in my case I have selected: WeMos-D1-WiFi-uno-based which include the WiFi ESP8266 built-in with the board, so no more effort is required.
- Temperature and Humidity Sensors : DHT11 is good enough.
- 5V Relay modules to use it to control devices on and off.
- Some connected wires to connect the sensor and relay module to the board.
- USB connector to connect the board to the laptop (in WeMos case, mobile connector USB will be good).
- Case for your board and components.
- Adapter for the Arduino board to connect it later with direct electricity
-  A PC/Laptop to be used to develop our platform.
- Soldering iron for production release of the device

Software Requirements
- Setup Arduino IDE so we can develop our Arduino Code.
- Google Cloud Tooling for Eclipse : here we will develop our back-end application code, deploy it to Google Cloud App Engine - Standard Environment (sandbox) which give us a chance to run it in a free tier.

- Optionally: Android Studio : to build the mobile application to be used as another way to control our smart home devices.

In subsequent posts we will go in depth in the complete solution.

Docker Overview

A container is packaged as an entire runtime environment: the service/app plus all dependencies, libraries, & configuration files needed to run it

Portable across environments & lightweight (share the OS)

The above image summarize the difference between container and VM, yet they can be combined and docker can be nested inside VM.

Different docker technoloy available such as Docker : www.docker.com, Mesos : http://mesos.apache.org/ and Kubernetes : https://kubernetes.io/

We will pick Docker to give high level functionality of it here.

Docker

Docker began as an internal project for the dotCloud organization. 

It was developed in-house and then later open sourced in 2013.

Enables you to:
Separate your applications from your infrastructure so you can deliver software quickly.
Manage your infrastructure in the same ways you manage your applications

As we can see Docker composed of Server (docker daemon) which expose the docker functionality via REST APIs, the docker command line client uses these REST APIs to communicate with the daemon service/server.

The main components as we can see is the Images, Containers, Network and data Volumes.
We can add to them the registries.

The following show the architecture and include the registry in the picture:

1)  Docker Images : Templates


An image is a read-only template with instructions “Dockerfile” for creating a Docker container. Often, an image is based on another image, with some additional customization.
You might create your own images or use those created and published by others in a registry.
When you change the Dockerfile and rebuild the image, only those layers which have changed are rebuilt.
This is part of what makes images so lightweight, small, and fast, when compared to other virtualization technologies.

Example of Dockerfile:

It is composed of 3 main parts, base image for that docker, different docker building steps including our application, finally the start command of that docker.
You should know that docker image is layered and Any RUN commands you specify in the Dockerfile creates a new layer for the container, this allow us to share the layers and build upon them which improve the usability of the containers and their layers.

2) Docker Registries: Templates Store
A Docker registry stores Docker images.
Docker Hub and Docker Cloud are public registries that anyone can use, and Docker is configured to look for images on Docker Hub by default.
You can even run your own private registry - “Docker Trusted Registry (DTR)”

You can push, and pull images from any Docker registry

A free https://cloud.docker.com/ account can be created where you can use it to store your docker images.

To use your Docker Cloud account:

docker login : will prompt for username and password

docker push : push to store any image in your 

docker pull : pull any image to your local machine

You can use:  docker search keyword to search for any docker image.

e.g. docker search oracle  ==> to search for Oracle images.

3) Docker Containers: Running instances
A container is a runnable instance of an image.
You can create, run, stop, move, or delete a container using the Docker API or CLI
You can connect a container to:
One or more networks
Attach storage to it
Capture a new image based on its current state.

4) Docker Network :
By default, Docker provides two network drivers:
Bridge (default) : limited to a single host running Docker Engine.
Overlay network : supports multiple hosts.
You can create your own network:
docker network create -d bridge my_bridge
To list existing networks: docker network ls
To add a docker into a network:  docker run -d --net=my_bridge …..
Optionally you can select the IP as well using --ip=ip_address (or --ip6=…)
To inspect network: docker network inspect my_bridge

5) Docker Volumes :

In addition to Docker Union File System which compose the Docker layers, Additional Storage can be mounted such as Data Volume :

Used to persist data, independent of the container’s lifecycle. 

Mounted during create or run of the docker using -v

Example : $ docker run -d -P --name web -v /webapp training/webapp python app.py

You can also mount existing host directory using the same –v

Example: $ docker run -d -P --name web -v /src/webapp:/webapp training/webapp python app.py

To list volumes: docker volume ls

Note: Shared storage can be used but need to pay attention to write operations to avoid data corruption.

6) Docker Swarm :

A swarm is a group of machines (physical or virtual) that are running Docker and have been joined into a cluster.
Contains Swarm Manager and Worker Nodes.
Uses several strategies to run containers:
“emptiest node” : which fills the least utilized machines with containers
“global” : which ensures that each machine gets exactly one instance of the specified container.
You execute: docker swarm init to convert this machine to Swarm Manager and then use: docker swarm join in other machines to join this cluster.

7) Docker Service :

A service only runs one image.

Described using: docker-compose.yml

Describing: what ports it should use, how many replicas, resources, etc.

docker stack deploy -c docker-compose.yml myapp

docker stack ps myapp

docker stack rm myapp

Example: Micro-service Example: Java REST App connect to Oracle DB

The following are the steps to create this example using docker command line:

Execution Steps:

//Build our Java Application Docker ...

docker build -t my_java_docker .

//Create Network for our Java & DB Dockers

docker network create -d bridge my_bridge

//Search for Oracle DB XE

docker search oracle

//Pull one of Oracle DB EX images (not official)

docker pull wnameless/oracle-xe-11g

//Now run the DB container in the my_bridge network ...

docker run --net=my_bridge -d -p 49160:22 -p 49161:1521 -e ORACLE_ALLOW_REMOTE=true wnameless/oracle-xe-11g

//List current running docker instances to get DB XE Instance ID

docker ps

//Access the DB Docker using bash and execute the required DB scripts.

docker exec -it "container id from the previous step" bash

//run SQLPlus to create our DB objects ..

sqlplus

//install DB objects

...

//Inspect my network to get the IP address of the DB container

docker network inspect my_bridge

//Run the Java container using the DB IP address as Environment variable in the Java docker container so it can connect to the DB successfully.

docker run -p 4010:80 --net=my_bridge -e DBAAS_DEFAULT_CONNECT_DESCRIPTOR=172.18.0.3:1521:XE my_java_docker

// or use this in env file in the format of ENV VAR=VALUE

docker run -p 4010:80 --net=my_bridge --env-file=./env.txt my_java_docker

That's just an introduction about Docker and Container Technologies.

For More information check Docker Documentation including samples and a lot of examples can be found at:
https://docs.docker.com/

Designing Scalable MongoDB Documents versus Relational DB Entities

One of the challenging to design MongoDB data model is the background knowledge of relational DB which will affect our ability to design optimal scalable data model structure.

In this post we will demonstrate a use case that is taken from the book; Instant MongoDB by Amol Nayak.

The use case is about Students enrolled in courses that taught by lecturers.
The relations can be summarized as following:

We have student use case where,
- Student enroll into courses (many to many)
- Each course can belong to many categories (one to many)
- Each course is delivered by many lecturers (one to many)
- Each course has content (one to one)
- Each course content is divided into parts (one to many)
- Each content part is related to assignments (one to many)
- Each student has assignment submission that is related to assignment (one to one)











Now to model this ER diagram for MongoDB documents, we need to do the following:

1) Think of the main documents that we have
The main document is a key player, well defined, and contains a lot of information that doesn't let it simply included in other documents.
We can think of Student, Course and Lecturer.

2) Embed Related Documents
We can see the Student embed his/her submissions while Course embed all other documents that is related to it such as catalog, content, assignment being all part of the course document.

3) Add reference to other documents (using id)
We can see the Student reference his/her courses.
Course reference the lecturers .

4) Add minimal information to the referenced documents
Select the information that is not frequently change and will be mostly needed in the application.
e.g. add course name in the referenced course in the student document (mostly will be required instead of go and query the course document, plus the course name is rarely change).
Also add lecturer name in the course document which will be mostly required and will change rarely as well but will prevent us from query the lecturer document to get the name with each course.

5) Revisit the documents 
To see if we can omit some documents and include them in one of the existing documents.
So for example if you decided to have a separate document for Course Category, at this step you'll see that the category has only name value so it is better to include it inside the course instead of reference it with id+name as it will cost us more information in that case.





 As we can see in the previous figure, we have identified 3 main documents with some embedded documents and we have selected the referenced documents, finally we have included the required minimal data in each of the referenced entities.
For example Course category has only static values, so we have included it entirely in our Course document and we didn't defined a separate document for it.
The same for Students submissions which reference the assignment but include all the required information so no separate document for it.
The other information that is related to course is also included in the course being part of the course document including the content parts, assignments, etc..

The challenge about the Lecture object here, the lecturer object has a lot of information about the lecturer that doesn't make any sense to put them in the course document and repeat them for different courses instead we can reference the lecturer document and define the minimal required information that we need to show or use it in our application, in this case lecturer name, the good thing about this information is not frequently change as well.

This is how to design documents in mongoDB for salable applications.