Wednesday, April 27, 2016

AngularJS Core Concepts

AngularJS is an Open Source Java Script Framework built by Miško Hevery and Adam Abrons in 2009.


Benefits:
- Easy to use
- Dependency Injection
- 2-ways Data Bindings
- Clear MVC : separate model, view and controller.
- Easy Testability
- Useful features such as directives, filters, etc..

Start to use:
Download Angluar.js file or angular.min.js (minified and obfuscated version to improve the page loading speed) and import it in your view (HTML) file.
Download from : https://angularjs.org/
Angular Modules:
Angular is designed based on modules which works as a container for all other angularJS components.
You can define any module using:
angular.module("module_name",[list_of_dependency_modules]
e.g. var myApp = angular.module("myApp",[]);
We can use this module inside our view or create anonymous application using ng-app
Angular controller:
Is a java script function that get executed in a specific scope.
For example we can add to our app a controller using:
var myController = function($scope){
    $scope.employees=[
        { "firstName" : "Osama", "lastName": "Oransa","picture": "/images/osama.png", "hiredate": new Date("June 29, 2000"), "salary": 1000, "tax": 0},
        { "firstName" : "Ahmed", "lastName": "Samir","picture": "/images/ahmed.png", "hiredate": new Date("November 06, 2000"), "salary": 1500, "tax": 0},
        { "firstName" : "Sameh", "lastName": "Alaa","picture": "/images/sameh.png", "hiredate": new Date("October 19, 2000"), "salary": 2000, "tax": 0},
        { "firstName" : "Toka", "lastName": "Ibrhaim","picture": "/images/toka.png", "hiredate": new Date("May 21, 2000"), "salary": 1400, "tax": 0}
        ];
}
myApp.controller("myController",myController);
So these 2 steps we have create a controller and added it to our application/module.
We can do this in a single step using:
myApp.controller("myController",function($scope){
    $scope.employees=[
        { "firstName" : "Osama", "lastName": "Oransa","picture": "/images/osama.png", "hiredate": new Date("June 29, 2000"), "salary": 1000, "tax": 0},
        { "firstName" : "Ahmed", "lastName": "Samir","picture": "/images/ahmed.png", "hiredate": new Date("November 06, 2000"), "salary": 1500, "tax": 0},
        { "firstName" : "Sameh", "lastName": "Alaa","picture": "/images/sameh.png", "hiredate": new Date("October 19, 2000"), "salary": 2000, "tax": 0},
        { "firstName" : "Toka", "lastName": "Ibrhaim","picture": "/images/toka.png", "hiredate": new Date("May 21, 2000"), "salary": 1400, "tax": 0}
        ];
});
$scope object is available to set data models and behavior so we can use them inside the view.
We can also delete from the scope object:
delete $scope.employees;

Defining behavior:
$scope.function_name = function(parameters) {
   ... code here
}
For example:
$scope.calculateTax= function(employee){
    employee.tax= employee.salary *0.2;
}
This add behavior and data model to our scope object where we can now use it from inside the view :
Example:
<html ng-app="myApp">
<head>
<script src="angular.js"/>
</head>
<body>
<div ng-controller="myController">
  <table>
      <tr ng-repeat="emp in employees">
        <td> Photo : <img ng-src="{{ emp.picture}}" />  </td>
        <td> First Name : {{ emp.firstName}}  </td>
        <td> Last Name : {{ emp.lastName}}  </td>
        <td> Hiredate : {{ emp.hiredate }}</td>
        <td> Salary : {{ emp.salary}}  </td>
        <td> Tax : {{ emp.tax}} <input type="button" value="Tax" ng-click="calculateTax(emp)"/> </td>
    </tr>
    </table>
</div>
</body>
</html>
We can have nested controllers and each one will be limited to its scope.
Directives, Events and Bindings:
In the previous example we have used ng-repeat to loop over collection.
We can use the following directives inside the loop:

$index     number     Number of the element      
$first     Boolean     This is true if the element is the first one      
$last     Boolean     This is true if the element is the last one      
$middle     Boolean     This is true if the element is in the middle      
$even     Boolean     This is true if the element is even      
$odd     Boolean     This is true if the element is odd   

we can also have nested loops in that case we can get the parent index using $parent.$index
We have also used ng-src for image to avoid using src directly which will fire 2 calls one before binding (which will fail) and another one after binding.
We used the {{ }} directive to bind values we can also use the directive ng-bing or ng-bindHtml to bind values.
We have used also ng-click to bind onClick event to that element, we can use also the following events:
ng-Blur, ng-Change, ng-Copy, ng-Cut, ng-DblClick, ng-Focus, ng-KeyPress, ng-KeyDown, ng-KeyUp, ng-Mousedown, ng-Mouseenter, ng-Mouseleave, ng-Mousemove, ng-Mouseover, ng-Mouseup, and ng-Paste
We can enable/disabled the element using ng-disabled="boolean value"
The same for ng-show and ng-hide which take boolean expression.
ng-if pevent the rendering of the element if the condition is not true.
We can also import certain parts in our view using ng-include directive.
e.g. <div ng-include="'mainMenu.html'"></div>

Two way Data Binding:
Change in the view update the model and change in the model update the view.
This can be used with input, textfield and text area, using ng-model
So for example in our previous example we can modify the Salary :
          <td> Salary : {{ emp.salary}} <input type="text" ng-model="emp.salary" /> </td>

Filters in AngularJS:
We can use Filters for 3 main use cases; Format, Sort or Filter the data.
To apply a filer using the pipe charachter "|"
e.g. {{ expression | filterName:parameter }}
5 filternames are available: lowercase, uppercase, number, currency and date.

LimitToFilter can be used to limit the number of rows or charachers in a String.
e.g. {{ expression | limitTo : limit : begin }}
$scope.rowLimit=3;
In the view:
Rows to Display: <input type="number" step="1" min="0" max="5" ng-model="rowLimit"/>
<br>
<table>
<thead>
<tr>
    <th>First Name</th>
    <th>Last Name</th>
    <th>Salary</th>
    <th>Tax</th>
</tr>
</thead>
<tbody>
<tr ng-repeat="emp in employees | limitTo:rowLimit:0">
<td> First Name : {{ emp.firstName |  upppercase }}</td>
<td> Last Name : {{ emp.lastName | lowercase }}</td>
<td> Hiredate : {{ emp.hiredate | date:"dd/MM/yyyy"}}</td>
<td> Salary : {{ emp.salary | number:2 }}</td>
<td> Tax : {{ emp.tax | currency:"$":1 }}</td>
</tr>
</tbody>
</table>

To sort data we use {{ orderby_expression | orderBy: expression : reverse }}
e.g. ng-repeat=" emp in employees | orderBy:'salary':false" or '+salary' or '-salary'

$scope.sortColumn = "firstName";
In view:
    Order By : <select ng-model="sortColumn">
                        <option value="firstName"> FirstName Asc</optoin>
                        <option value="-firstName"> FirstName Desc</optoin>
                        <option value="+salary"> Salary Asc</optoin>
                        <option value="-salary"> Salary Desc</optoin>
                    </select>

    <br>
    <table>
    <tr>
        <th>First Name</th>
        <th>Last Name</th>
        <th>Salary</th>
        <th>Tax</th>
    </tr>
    <tr ng-repeat="emp in employees | orderBy:sortColumn">
        <td> First Name : {{ emp.firstName |  uppercase }}</td>
        <td> Last Name : {{ emp.lastName | lowercase }}</td>
        <td> Hiredate : {{ emp.hiredate | date:"dd/MM/yyyy"}}</td>
        <td> Salary : {{ emp.salary | number:2 }}</td>
        <td> Tax : {{ emp.tax | currency:"$":1 }}</td>
    </tr>
    </table>
To sort using column header:
    $scope.reverseSort=false;
    //construct sorting column and reverse flag
    $scope.sortDate= function(column){
      $scope.reverseSort = ($scope.sortColumn == column) ? !$scope.reverseSort : false;
      $scope.sortColumn = column;
    }
    //add function to return css to show arrow beside the column name
    $scope.getSortClass= function(column) {
     if($scope.sortColumn== column) {
        return $scope.reverseSort ? 'arrow-down' : 'arrow-up'
     } else return '';
    }

In the view now we need to define the CSS for the arrow-down and arrow-up.
        <th ng-click="sortDate('firstName')"> First Name <div ng-class="getSortClass('firstName')"/> </th>
        <th ng-click="sortDate('lastName')"> Last Name <div ng-class="getSortClass('lastName')"/> </th>
        <th ng-click="sortDate('salary')"> Salary <div ng-class="getSortClass('salary')"/> </th>
        <th ng-click="sortDate('tax')"> Tax <div ng-class="getSortClass('tax')"/> </th>
<tr ng-repeat="emp in employees | orderBy:sortColumn:reverseSort">
..
</tr>

Finally in css file:
th{
      cursor: pointer;
  }
  .arrow-up {
      width: 0;
      height: 0;
      border-left : 5px solid transparent;
      border-right : 5px solid transparent;
      border-bottom : 10px solid black;
      display: inline-block;
  }
  .arrow-down {
      width: 0;
      height: 0;
      border-left : 5px solid transparent;
      border-right : 5px solid transparent;
      border-top : 10px solid black;
      display: inline-block;
  }
This will enable the sorting using the table headers.

Form Validations:
To mandate a field: ng-required="true"
For min and max we can use
    ng-minlength="6"
        ng-maxlength="10"
We can also add reqular expressions using: ng-pattern
ng-pattern="/^[A-z]+$/"  to restrict it to alpahbets only
If we have form like <form name="personalForm"> and a field like:
<form name="personalForm">
        <div ng-show="personalForm.name.$dirty && personalForm.name.$invalid">Something went wrong! <br>
        <span ng-show="personalForm.name.$error.required">
        You must enter the first name!<br>
        </span>
        <span ng-show="personalForm.name.$error.minlength">
        The name must have at least 6 characters!<br>
        </span>
        <span ng-show="personalForm.name.$error.maxlength">
        The name must have at most 10 characters!<br>
        </span>
        <span ng-show="personalForm.name.$error.pattern">
        The name must only include alphabets!<br>
        </span>
        </div>
        <td> <input type="text" name="name" ng-required="true" ng-model="name" ng-minlength="6" ng-maxlength="10" ng-pattern="/^[A-z]+$/" />
        <input type="button" value="Submit" ng-disabled="personalForm.name.$invalid"/>
  </form>
now we can use attribute to enable or disable the submit button:
        <input type="button" value="Submit" ng-disabled="personalForm.name.$invalid"/>

We can also show alert using the field attribute $invalid
<div ng-show="personalForm.name.$dirty && personalForm.name.$invalid">Something went wrong! <br>


Search Filter:
Same concept using filter:key_word:exact_match
e.g.
First Name : <input type="text" ng-model="searchText.firstName" placeholder="search first name"/>
    Last Name : <input type="text" ng-model="searchText.lastName" placeholder="search last name"/>
    <input type="checkbox" ng-model="exactMatch"> Exact Match

And in the repeat section:
    <tr ng-repeat="emp in employees | filter:searchText:exactMatch">

Custom Filter:
To create custom filter all what we need to do is to define a filter function that return a function:
myApp.filter("genderFilter",function() {
    return function(gender) {
        switch (gender) {
            case 1: return "Male";
            case 2: return "Female";
            default: return "N/A";
        }
    }
});

Now from inside the view use it as :
<td> {{ emp.gender | genderFilter }} </td>


AngularJS Service:
To call a REST service to retireve the data, all you need is the end point and know the
structure of your response data which is mostly JSON.
For example if we have the following end point that retrieve all employees from the DB as a REST service with JSON structure:
http://localhost:8090/ords/my_hr/my_emp/

We can call it using a controller:
myApp.controller("myController2",function($scope,$http){
        $http.get('http://localhost:8090/ords/my_hr/my_emp/')
            .then(function (response) {
                $scope.employees2 = response.data;
            });
});
We have used get HTTP method, we can use POST, DELETE, PUT, etc.
 Alternatively we can use the following way where we send configuration object as a parameter to the $http service:
myApp.controller("myController2",function($scope,$http){
        $http(
            {
            url:'http://localhost:8090/ords/my_hr/my_emp/',
            method: 'get'
             })
            .then(function (response) {
                $scope.employees2 = response.data.items;
            });
});
We can also supply another function for errors:
$http({
            url:'http://localhost:8090/ords/my_hr/my_emp/',
            method: 'get'})
            .then(function (response) {
                $scope.employees2 = response.data.items;
                $log.warn(response.data);
            }, function(reason){
                $scope.error= reason.data;
                $log.error(reason.data);
            });
Where we can get the reason object of such error.
We can also define these functions separately and then assign them in the then() method
e.g. var function successCallBackFunction= function (response){ ... } &
var function errorCallBackFunction= function (reason){ ... } &
Now change then method into :
.then(successCallBackFunction,errorCallBackFunction)

NOTE: if you are using Oracle REST Data Service - ORDS you need to retireve the data using response.data.items;

This will invoke the service and set the data in the scope once the response is available.
In the view you can consume it as usual:
      <tr ng-repeat="emp in employees2">
        <td> {{ emp.first_name}}  </td>
        <td> {{ emp.last_name}}  </td>
        <td> {{ emp.hire_date | date:"dd/MM/yyyy"}}</td>
        <td> {{ emp.salary}}</td>
        <td> {{ emp.email}} </td>
    </tr>

That's the simplest way to interact with REST services.

Use Logging Service:
Simply we need to import the log service in controller parameters
e.g. myApp.controller("myController2",function($scope,$log){

and we can then use it as $log.info('message here'); or use other log messages types such as debug, warn and error.

Building Your Own Service:


Service in AngularJS are JS objects that could contain atributes or functions for example $scope, $http and $log services, to build your own service you need to simply define the service and its attributes/functions so you can use it inside your application.
This should promotes usability inside your application.
For example we can define a new service named : service_name

myApp.factory('service_name',function() {
    return {
        //the JS object here attributes and functions
        helloWorldFunction : function(input) {
            return "Hello "+input;
        }
    };
}
To use it in any controller, all you need is to reference your new service
e.g.
myApp.controller("myController2",function($scope, service_name){
     $scope.greeting= service_name.helloWorldFunction('Osama Oransa');
}

Useuful Tutorial:
One of the most useful AngularJS tutorials is the following one:
https://www.youtube.com/watch?v=zKkUN-mJtPQ&list=PL6n9fhu94yhWKHkcL7RJmmXyxkuFB3KSl
where a lot of the content in this post is based on that tutorial.


Thursday, February 11, 2016

Big Data & Hadoop Map Reduce

What is Big Data?

Big data as the name suggest refer to the data with the following 3 characteristics:
1- High Generation Rate i.e. High Velocity of data generation
2- Big data size i.e. big Volume of data
3- Many sources for the data i.e. Variety of data sources.
These 3 Vs = Velocity (speed) Volume (size) Variety (sources) represents what is named the big data, such data is usually semi-structured data from different sources.

Apache Hadoop:
Apache Hadoop is an open-source framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built.


 
So we are discussing here 2 concepts in hadoop distributes storage and distributed processing.

1) Distributed Storage: 
In Hadoop Data stored in HDFS : Hadoop Distributed File System where each file is splitted into blocks (default size of each block is 64MB) blk_1, blk_2, etc..
Each block is replicated 3 times in Data Nodes (for data redundancy) (this means if u have 100MB of data you'll need 300MB of storage).  and if any node failed, hadoop will auto-replicate the data into another data node.
The Name Node determines the location of different parts of each file stored in HDFS.
Name node is stored usually in NFS (Network file system) for higher reliability and it runs also on active-standby node model where the standby node can take over if active node failed.
2) Distributed Processing:
We have 2 concepts here; the mappers and reducers for distributed data processing.
- Each Mapper works in small set of data and produces intermediate records in a (key, value) pairs format.
- Data go through Shuffle step which move the data into reducers and sort them (optionally)
- The Reducers can reduce the data into format (key, values).
- Finally if we have multiple reducers additional step is required to merge and sort the keys to produce the final required results.

Job Tracker is responsible for splitting the work into different mappers and reducers.

In each data node, a Task Tracker exist which will usually assign the existing data block to the mapper in the same node to reduce the required network traffic but in case the current mapper is busy (for the 3 copies of the data block) the task tracker can delegate the task into a different data node mapper and stream the block into that node but this happens rarely.

Combiners can do some reduction to reduce the data moved over the network.







 
Hadoop Common Design Patterns:
1) Filtering Patterns
e.g. Sampling or Top-N list

Filter the records, do not change them, keep some and discard some.
The output is subset of the original data-set.

Types:
- Simple filter e.g. function
- Bloom Filter
- Sampling Filter
- Random Sampling
- Top-n filter.
  Each mapper find the top-n list and the reducer produce the final top-n from these lists.


2) Summarization Patterns
e.g. Counting, Min/Max, Statistics, index, etc.
Types:
- Inverted Index (reverse index for faster search)
- Numerical Summarization
    e.g. count, min/max, first/last, mean, median, average, etc.
Combiners are important here.

3) Structural Patterns
e.g. Combining data sets.
Structural to Hierarchical Pattern
- Moves the structured data in RDBMS into Hadoop but hierarchical
the data must be linked in the source using FK and must be structured in row-based.

4) Others such as Organization, I/O, ...etc.





For More Details

This is just a nutshell about Big Data and Hadoop Map Reduce and as of the Udacity

Intro to Hadoop and MapReduce


Sunday, January 31, 2016

NodeJS Quick Guide - Part 2

In this part we will use the express web application to develop a sample web application.

How to customize the application:
1) Change the port
We can change the default port to a new port or better use the runtime arguments to set it.
For example:
if (process.argv[2]!=null){
    var port = normalizePort(process.argv[2]);
} else {
    var port = normalizePort(process.env.PORT || '8000');
}
app.set('port', port);

This will let us use the application by providing the port as an argument as following:
set DEBUG=mytestapp:* & npm start 8002

2) Add images in the images folder:
Download any HR logo image and place it in the images folder.

3) Add or Edit Routes
We can edit the current pages or create a new routes.
Let's create header and footer pages and place all these files in the views folder
1) header.jade 
<html><head><title>#{title}</title></head><body><center><h1>#{title}</h1><img src="/images/hr-logo.jpg" width=200 height=100><hr><br>

2) footer.jade
<hr><div> (c) Copyright 2016 </div></center></body></html>

Now modify index.jade to be as following:
include ./header.jade
block content
  h1= title
  p Welcome to #{title}
p Enter an Employee number to see his/her data
</p><form name='users' action='/users' method='get'><input type='text' name='emp_num' /><input type='submit' value='Submit' /></form>
include ./footer.jade


As you can see in these files, we have mixed the html tags with jade syntax e.g. <p> or p
We use the format #{ ... } for variables.
We have included the header and footer files using include statement.
The form in index.jade submit the user inputs to the /users

The index.js should be modified a little bit to include the correct page title:
var express = require('express');
var router = express.Router();
/* GET home page. */
router.get('/', function(req, res, next) {
  res.render('index', { title: 'HR Express' });
});
module.exports = router;


Let's now edit the user.js file to capture this input field and route it into details page.

var express = require('express');
var router = express.Router();
/* GET users listing. */
router.get('/', function(req, res, next) {
  res.render('emp', { title: 'HR Express', number: req.query.emp_num, name: 'Osama Oransa', position: 'PaaS Expert' });
});
module.exports = router;


You can see the query parameter is captured using req.query.parameter_name we have also set some variables, these variables should typically comes from the database tables.

Add emp.jade view with the following content:
include ./header.jade
block content
  h1= title
  p Welcome to #{title} - Employee Details
p Here is the Employee Data
P Number : #{number}
P Name : #{name}
P Position: #{position}
<br>
p
<a href='/'>Back</a>
include ./footer.jade


4) Test the application
We can now test the application by executing our command:
set DEBUG=mytestapp:* & npm start 8002

In the home page enter any employee number and click submit to view the details page.










5) Using Session object
To use the session object we need to install the following modules: connect and cookie-session.
npm install connect
npm install cookie-session


In app.js add the following to instantiate the session using cookies:
var connect = require('connect');
// store session state in browser cookie
var cookieSession = require('cookie-session');
app.use(cookieSession({
    keys: ['secret1', 'secret2']
}));

You can change the session keys to any unique keys.
Now in index.js modify the code to store a random guest name in the session.
var express = require('express');
var router = express.Router();
/* GET home page. */
router.get('/', function(req, res, next) {
  if(req.session.guest==null) req.session.guest='Osama '+random(1,10);
  res.render('index', { title: 'HR Express','guest':req.session.guest });
});
module.exports = router;
function random (low, high) {
    return Math.floor(Math.random() * (high - low + 1) + low);
}

In users.js add the parameter as well to the variables:
/* GET users listing. */
router.get('/', function(req, res, next) {
  res.render('emp', { title: 'HR Express','guest':req.session.guest, number: req.query.emp_num, name: 'Osama Oransa', position: 'PaaS Expert' });
});


Finally in the header.jade add a p for the guest name just above the image:

<h1>#{title}</h1>
p Welcome #{guest}
<br>

6) Test the application
Now test the application using 2 different browsers to see the effect of the session.

You can see the guest name is now below the title, test using different browsers.

7) Connect to the DB
Now to connect the database we have many options, we can connect on SQL or NO-SQL database for example to use Oracle Database you need to modify the code to query the database.
Oracle provides a DB driver for NodeJS can be found in the following URL:
https://www.npmjs.com/package/oracledb
To Install it, you'll need the following steps:
Prerequisites:
    - Python 2.7
    - C Compiler with support for C++ 11 (Xcode, gcc, Visual Studio or similar)
    - The small, free Oracle Instant Client libraries if your database is remote. Or use a locally installed database such as the free Oracle XE release
    - Set OCI_LIB_DIR and OCI_INC_DIR during installation if the Oracle libraries and headers are in a non-default location
Installation:
Run:
 npm install oracledb
This will install it from the NPM registry.

Let's now modify the code to get the employee details from the DB (Oracle HR sample schema) as following:

var oracledb = require('oracledb');
oracledb.getConnection(
  {
    user          : "hr",
    password      : "hr",
    connectString : "localhost/XE"
  },
  function(err, connection)
  {
    if (err) { console.error(err.message); return; }
    connection.execute(
      "Sselect first_name||' '||last_name name, job_title "+
      " FROM employees,jobs"+
      " WHERE employees.job_id=jobs.job_id"+
      " and employee_id = :id ",
      [110],  // bind value for :id 

{outFormat: oracledb.OBJECT}, // outFormat can be OBJECT or ARRAY. The default is ARRAY
       function(err, result)
      {
        if (err) { console.error(err.message); return; }
        console.log(result.rows);
      });
  });

 We need to replace the [110] with our employee parameter: req.query.emp_num,
 And pass the result into our emp.jade page to render the employee details.
result[0].name & result[0].job_title
We need also to route the errors into error pages.
You can see here a lot of examples of how to use Oracle driver:
https://github.com/oracle/node-oracledb/tree/master/examples

This is a simple application to show us how to use some of the NodeJS features in nutshell.

NodeJS Quick Guide - Part 1

NodeJS developed by Ryan Dahl in 2009 which runs using Chrome JavaScript engine plus some libraries for asynchronous network and I/O modules.
It targets building high performance and scalable server and client applications. Node.js uses Just-in-Time compilation with the V8 JavaScript Engine.





1) Installing NodeJS
The quickest and easiest way is to install it from the nodejs.org web site.
https://nodejs.org/en/download/

Once installed from the command line you can write node or npm to ensure everything is fine.

2) NodeJS
Uses the  java script language where you can write loosely typed event based asynchronous code.
Loosely typed: means the variable types can be determined at runtime not compilation time as in Java which gives power but could lead to potential errors as you already lost the compiler error scanning.
Event-based: means you need to fire action and wait for the response/output in a function that process this output so instead of waiting the DB operation to finish fire the query and wait the response in a function, this unlike the thread based languages such as Java where you need to wait for the response , the gain is nullifying the context switching and simplify the code, which gives higher performance in throughput of the application.
So every function that requests IO has a signature like function (... parameters ..., callback) and needs to be given a callback that will be invoked when the requested operation is completed i.e. Continuation-passing style.

Note that Node is not suitable for high computation operations and it needs good design to process such operations without impacting the application.
In spite NodeJS is a single threaded, and the potential feature for fork child-processes or threads is rolled back, some modules provide a way to run some threads for high computation consuming operations one example is Threads a gogo
https://github.com/xk/node-threads-a-gogo

3) NodeJS Modules:
Node is based on modules which is similar concept to packages in Java, the modules are used by the statement:
var http = require('http');    ==> This means the http variable will be used to access the exposed functions in the http module.
The http module as an example here can expose different functions now by the following statement: exports.function_name e.g. :
exports.stop = function(){
console.log('Inside the stop function');
}


One of the big issues in JavaSciprt is the global variables, using the modules and exports you can control what you need to expose and any global variable is now considered local to that module and won't interfere with any similar name variable in your application.

To install modules that you need in your application, you need to use npm where you specify if you need to install the module for your application or globally for all applications by using:

npm install -g express ==> this will install express a web application framework based on the open source connect project.
if you omit -g you will install the packages locally for your application.
The modules are loaded into the node_modules folder either in your application or globally.

4) Running a hello world application
The following sample application uses the http module to start a server:
var http = require('http');
exports.start = function(){
server=http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello, World!\n');
}).listen(8082, '127.0.0.1');
console.log('Server is now running: http://127.0.0.1:8082');
}

Save it into a file name; app.js
now run node
> node
> var app=require('./app');
> app.start();

Server is now running: http://127.0.0.1:8082

Open the browser on the port 8082 using localhost:8082 you can see hello world message is printed out.
In our application we have exported start(); function() and from the node interpreter we have loaded our application module and executed this exposed start() function.
Our application uses the http module to run the http server.
You can run wrap it in another application
var app = require('./app');
app.start();

Save it as app-runner.js and now run it using:
node app-runner

another way is to use the package.json file which describes your application and dependencies
create the file package.json and add the following JSON content:
{ "name": "AppRunner",
  "version": "1.0.0",
  "scripts": {
    "start": "node ./app-runner"
  },
  "dependencies": {
  }
}

NOTE: to create the file you can execute npm init to create this package.json file.
It simply shows the application name, version, start script and any dependencies should be added to that section.
Now we can run the application using the command
npm start ==> which will search for that file and run the start command.
also npm install ==> will install all the required modules in the dependencies section in the local application directory.
Finally npm list shows the installed modules for this application.

5)  Using express web application framework
Express is a Web Application Framework for node:
http://expressjs.com/en/starter/generator.html
- Install express and express-generator
npm install express -g
npm install express-generator -g

Now use it to create a sample web application
express mytestapp   ==> This will generate the web application framework in the mytestapp folder
cd mytestapp ==> you can see the main application app.js and package.json where you need to add any future dependencies.
npm install ==> to install all required modules locally.
set DEBUG=mytestapp:* & npm start ==> in windows to start the application server
Test the application in the browser using localhost:3000

The following contains the APIs Guide for using Express:
http://expressjs.com/en/4x/api.html

The generated sample application contain the following main components:

1) Package.json file
In that file we can see the start script is
"scripts": {
    "start": "node ./bin/www"
  },

If we opened this file /bin/www , we can see that it uses the app.js, do the port configurations and the initialization of http server, which means if environment variable PORT is set, use it otherwise use the port 3000.
var port = normalizePort(process.env.PORT || '3000');
app.set('port', port);

We can change this port 3000 into any port e.g. 8082

2) View folder for views
We can add our views/pages here.
app.set('views', path.join(__dirname, 'views'));
The view engine to parse the views is specified as:
app.set('view engine', 'ejs');
or
app.set('view engine', 'jade');
You can use both options to build your view files.
Here is a quick guide for jade: http://jade-lang.com/reference/plain-text/
And here is a quick guide for ejs: http://www.embeddedjs.com/getting_started.html

3) Public folder for static contents:
This folder for storing the images, stylesheets and client side java script files.

4) Router to router the requests.
var routes = require('./routes/index');
var users = require('./routes/users');
app.use('/', routes);
app.use('/users', users);

one is index.js for the / path which is placed in the routes folder and another one is users.js for the path /users and is placed as well in routes folder.
You can replace content in both files to see the impact of opening both URLs http://127.0.0.1:8000/users and http://127.0.0.1:8000/

In part II, we will continue the application exploration by adding different pages and using parameters, forms, sessions and access the database to retrieve data.

Wednesday, January 27, 2016

App Engine's Datastore


     App Engine's datastore is built on top of Bigtable. To use App Engine we need to create account using our gmail, create identifier/key for our application and finally create our application in Eclipse.
You can install Google App Engine Plugin in any Eclipse e.g. in Luna use the following URL: https://dl.google.com/eclipse/plugin/4.4

This represents all Google tools, select the App Engine tools.

To use Google datastore from inside our Java code we can use one of the following three methods:

A) Using Java Data Objects (JDO) 

JDO is a standard interface for accessing databases in Java, providing a mapping between Java classes and database tables.
Let's explore how to use JDO to access datastore:
We can follow the following simple steps:

1) Create the Entity Class(es) :
@PersistenceCapable(identityType=IdentityType.APPLICATION)
public class PersonLocation implements Serializable {
private static final long serialVersionUID = 1L;
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
public Long id;
@Persistent
public int personId;
@Persistent
public double lat;
@Persistent
public double lon;
@Persistent
public double accuracy;
+ settings, getters and constructors.

2) Get the persistent manager factory:

PersistenceManagerFactory pmfInstance = JDOHelper
.getPersistenceManagerFactory("transactions-optional");

3) Get PersistenceManager from the factory and use it for CRUD operations:
PersistenceManager pm = pmfInstance.getPersistenceManager();

4) Create Entity :
Create an object from the entity and persist it.
PersonLocation location = null;
try {
location=new PersonLocation();
location.setPersonId(entityId);
location.setLat(lat);
location.setLon(lon);
location.setAccuracy(accuracy);
pm.makePersistent(location);
System.out.println("id="+location.getId());
} finally {
pm.close();
}
Once the entity persistent you can get the id.

5) Get Entity:
Either use the getObjectById as in the previous methods or execute datastore query to retireve the requied data.
location = pm.getObjectById(PersonLocation.class, id);

Or use the Query object, the query retrieves all entities of the given kind that satisfy all of the given filters, sorted in the specified order.
    public List getAllLocationsForEntity(int entityId){
PersistenceManager pm = pmfInstance.getPersistenceManager();
String queryStr = "select from " + PersonLocation.class.getName();
Query selectQuery=pm.newQuery(queryStr);
selectQuery.setFilter("personId=="+entityId);
return (List) selectQuery.execute();
}

6) Update Entity: 
Get the entity, update the entity and persist the updates.
try {
pm.currentTransaction().begin();
// We do lookup for the required id
location = pm.getObjectById(PersonLocation.class, id);
location.setPersonId(entityId);
location.setLat(lat);
location.setLon(lon);
location.setAccuracy(accuracy);
pm.makePersistent(location);
pm.currentTransaction().commit();
} catch (Exception ex) {
pm.currentTransaction().rollback();
throw new RuntimeException(ex);
} finally {
pm.close();
}



7) Delete Entity: 
Get the entity and delete it.
try {
pm.currentTransaction().begin();
// We do lookup for the required id
location = pm.getObjectById(PersonLocation.class, id);
pm.deletePersistent(location);
pm.currentTransaction().commit();
} catch (Exception ex) {
pm.currentTransaction().rollback();
throw new RuntimeException(ex);
} finally {
pm.close();
}



B) Using JPA

Similarly to using JPA:
1) In persistence.xml
            org.datanucleus.api.jpa.PersistenceProviderImpl
2) Get the EntityManagerFactory:
 private static final EntityManagerFactory emfInstance =
        Persistence.createEntityManagerFactory("transactions-optional");
3) Get the EntityManager from the EntityManagerFactory:
EntityManager em = emfInstance.createEntityManager();
4) Annotate your entity:
@Entity
public class Employee {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Key key;
    private String firstName;
    private String lastName;}
... setters, getters and constructors
 }
5) Use the entity manager to do the JPA CRUD operations. 
   

C) Using the low-level Datastore API, or one of the open-source APIs developed specifically for Datastore, such as Objectify. 

This is recommended by Google as JDO and JPA was designed for use with traditional relational databases, and so has no way to explicitly represent some of the aspects of Datastore that make it different from relational databases, such as entity groups and ancestor queries. This can lead to subtle issues that are difficult to understand and fix.
https://github.com/objectify/objectify

Sample Code:
@Entity
class Car {
    @Id String vin; // Can be Long, long, or String
    String color;
}

ofy().save().entity(new Car("123123", "red")).now();
Car c = ofy().load().type(Car.class).id("123123").now();
ofy().delete().entity(c);

Sunday, December 13, 2015

Simplified Guide To Use MongoDB

MongoDB is one of the No-SQL database engines, it is Document Oriented Database.
Documents can represent any structure that can fit JSON format so for example the following is a car document :
{
        "_id" : 6,
        "Car" : "BMW",
        "Model" : "S5",
        "Price" : 3455,
        "Color" : "Blue",
        "Date" : "09/10/2015",
        "Full Options" : "YES",
        "Level" : 1
}


Simplified Steps to use MongoDB:

1) Install the database locally...

Download it and install it.
From the command line execute: mongod and specify the location of the database files (any valid path so it can be used for storing database files).
mongod --dbpath c:\MongoDB\Database
All database command lines exist in the bin folder under Server directory.

2) Create any test data ..

You can create .csv file or .json file the simplest and quickest way is to use .csv file.
For example:
_id,Car,Model,Price,Color,Date,Full Options,Level
1,Mercedes,E-Class,12121,Red,12/12/2015,YES,1
2,Renault,Note,1223,Blue,09/09/2014,NO,2
3,Daihatsu,Terios,2333,Gray,10/10/2010,NO,2
4,BMW,S3,3988,Black,05/10/2008,YES,1


As you can see the _id is the primary key in MongoDB (if you didn't specify it, MongoDB will generate it for these entries.

3) Import test data

From the command line run the following commands:
mongoimport -d catalog -c cars --file cars.csv --type csv --headerline
This will import our file into Catalog database (similar to Schema in SQL DB) in a collection named cars (similar to DB table in SQL DB)
Note: if you import json file you don't need to specify the --type or --headline

 4) Manipulate MongoDB data 

Execute mongo from the command line ...
The default will connect to the localhost running instance ...
To connect to our database (schema) that we have created for cars, execute:
 use catalog
Now we can see all existing collections i.e. tables by executing:
 show collections
which will display only "cars" collection.
To deal with "cars" we can use the following:
 db.cars.count()    
To retrieve the count of the rows in this collection.
 db.cars.find()
To select all cars records
 db.cars.find({"Level":1,"Color":"Black"})
To select cars with level 1 and color is black.
 db.cars.find({"Price":{$gt:2000}})
To select cars with price more than 2000.
 db.cars.find({"Price":{$lte:3000},"Color":/l/}).pretty()
To select cars with price less than 3000 and color contain a "l" letter inside and then format it in JSON readable format.
 db.cars.find({"Price":{$gte:1000}}).sort({"Price":1}).skip(1).limit(2).pretty()
To select all cars with price more than or equal 1000 and sort the return by price ascending and limit the output to 2 only after skip the 1st results (we can use -1 to sort the result descending).
Note: You can use findOne() instead of find() to return the 1st match only.
To insert a new record :
db.cars.insert({ "_id" : 6, "Car" : "BMW", "Model" : "S5", "Price" : 3455, "Color" : "Blue", "Date" : "09/10/2015", "Full Options" : "YES", "Level" : 1 })
 To update the price of this inserted record we can use:
db.cars.update({_id:6},{$set:{"Price":5432}})
To delete a row in the collection use:
db.cars.remove({"_id":5})
To drop the whole collection: (don't execute it now)
db.cars.drop()To analyze a query performance we can execute explain plan as following:
db.cars.find({"Color":"Red"}).explain()
We can create index to speed up the query by the following:
db.cars.ensureIndex({"Color":1}) 
This means we will create index for the Color field in ascending manner.
Re-execute: db.cars.find({"Color":"Red"}).explain() to see the effect of this index.

5) Use MongoDB from inside Java Code


1- Add Mongo required drivers to your project classpath ...
 - mongodb-driver-3.0.4.jar
 - mongodb-driver-core-3.0.4.jar
 - bson-3.0.4.jar

2- Write your code to connect to the Database as following:
MongoClient mongoClient = new MongoClient();
This connect to a local running mongoDB on a default a port.
MongoDatabase db = mongoClient.getDatabase("catalog");
This connect to our catalog database.
FindIterable iterable = db.getCollection("cars").
                find(new Document("Level", 1)).
                sort(Sorts.ascending("Price"));
        iterable.forEach(new Block() {
            @Override
            public void apply(final Document document) {
                System.out.println(document);
            }
        });

This query our cars collection for Cars with level 1 and sort them ascending by Price, iterate over the output and print each Document (i.e. JSON string).
 To insert an entry use the same way:
String json = "{ \"_id\" : 5, \"Car\" : \"Kia\", \"Model\" : \"Picanto\", \"Price\" : 777, \"Color\" : \"Blue\", \"Date\" : \"09/09/2011\", \"Full Options\" : \"NO\", \"Level\" : 3 }";
db.getCollection("cars").insertOne(Document.parse(json));

You can convert Bean objects into JSON and vice versa using any existing APIs..

This is simplified overview that covers most of MongoDB in nutshell and let you master how to deal with this database in just 5 minutes.