Using Diffie–Hellman Key Exchange to Secure RESTful WS

Osama M. Oransa
Arab Academy for Science, Technology & Maritime Transport


Abstract
Representational State Transfer (REST) [1] is an architectural style for building large-scale distributed information networks, the most famous of these being the World Wide Web [12]. RESTful web services have an issue with its security model, lacking a standard way to secure it, in the same time its security relies in the underlying protocol. [8]

Existing solution to secure it, is to use https instead of http, the issue with that approach is that , this is a transport layer protocol so it doesn’t provide authentication or authorization yet it secure the end to end channel between both parties.

In the proposed solution, Diffie–Hellman key exchange is used to generate a session key to be used within the interactions to secure the requested resources and encrypt the resource representation or parts of it; in this solution a pluggable filter is introduced with the advantages of avoid any change in the existing code and secure only the needed resources.

The suggested solution protect the resources and provide both authentication and authorization, this would fix the security issues related to RESTful web services but results in stateful filter that contradict with the basic concepts of this technology.

Keywords
RESTful Web Services; Diffie–Hellman Key Exchange; Web Services Security.



I. Introduction
RESTful web services emerged as a scalable way of doing web services yet there is not mature as the traditional SOAP-based web services that has a lot of security standards including WS-SE??.

Some suggested solutions to secure the RESTful web services includes using https instead of http, the issue with that approach is that , this is a transport layer protocol so it doesn’t provide authentication or authorization yet it secure the end to end channel between both parties, anther way is to send authentication data with each request, this could provide authentication , but subjected to other attacks in case as man-in-the-middle attack, another way is to send a token granted from token generating service like OAuth[11] this have the same previous solution [9] most successful if used in trusted environment like organization intranet [10].
In the proposed solution, Diffie–Hellman key exchange is used to generate a session key to be used within the interactions to secure the requested resources and encrypt the resource representation or parts of it; in this solution a pluggable filter is introduced with the advantages of avoid any change in the existing code and secure only the needed resources.


II. Background
REST refers to Representational state transfer, a terminology first mentioned by Fielding, Roy Thomas in his Doctoral dissertation “Architectural Styles and the Design of Network-based Software Architectures” in the year of 2000. [1]

The concept is known since the evolution of the internet but it was first time to be described and this make sense as Fielding was one of the authors of http protocol specifications (1.0 and 1.1)[4][5], the 1st interest in this architecture appears once the programming languages support the REST feature; Java in 2007/2008 [2]

In spite the wide adaptation of RESTful web services in the public exposed services; it is still has some areas that need to be enhanced in compare to SOAP-based web services where there are a lot of existing standards referred as WS-* standards (e.g. WS-Addressing, WS-Security); one of these areas is the web service security where it lack any standard way to implement it and actually the only existing standard way is to use https protocol as a transport layer security protocol.

For message-level security, it’s using HTTPS seems enough, but for other capabilities such as signatures, encryption, or federation cannot be supplied by HTTP alone. [3]
One existing solution is to have authentication token by a secure token service then use the token to access this RESTful web services. [3] The standard description of the RESTful web services is Web Application Description Language (WADL). [6]

Diffie–Hellman key exchange method - first published by Whitfield Diffie and Martin Hellman in 1976 - allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure communications channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher. [7]

III. Proposed Model
The proposed solution here utilize the Diffie–Hellman key exchange to generate a session key using the normal http protocol without the need for any secure tunneling so the client and the server can both have the same key and use it to encrypt the parts the need to be secure in the REST calls.
For example the resource URL can be encrypted , and only the server can identify it using the session key, also the returned resource could be also encrypted (partially or completely) so only the client requested this resource can decrypt it using the session key.
This is implemented using intercepting filter that authorize the client and validate the resource access rights, the main advantage of being implemented as a filter is that it can be configured per the needed resources so you can still provide some without security, also no need to change the RESTful WS implementation as it would be just another layer over it so its pluggable, and you don’t need the code to be repeated being centralized location for these requests.
The server will invalidate the key after a configurable time (session time) in that case the client is required to generate a new session key to interact, this filter here represents the only part of the system that behave as stateful not stateless as the RESTful web services behavior.



Figure 1: Represents the different component in the system and different message types between the client and our proposed RESTful System.

As shown in the above diagram the system has client data represents the needed information which are the identification and 2 selected values to generate the session key, another component to authorize each user to different resources, describes which component could be interacted by the client and to which level (read, create, update and delete).
The client can interact in 2 steps 1st to get a session key then to manipulate the resources (step 1 and 2 in the diagram), of course the session key will expire after certain period of time so the client call need to be wrapped to get a new session key if the call return session key expired exception or to have some sort of scheduler run to get a new session key, the session key here represent a resource in the system like any other resources so the user need to interact with it on this concept; it is a good way to increase the security over the connection as without the need of https the channel is dynamically secure, yet the user can choose if the resource is needed encrypted or not according to the needs plus the ACL that determine additional attribute if this resource is allowed to be interacted non-encrypted.

IV. Results
The system is used to control integration layer between the 2 tiers , the system provide a good way to secure RESTful web services without the need to use https, the security is dynamically obtained using a session key generated by Diffie–Hellman key exchange method, also the system provide a way to authorize the access on different resources.
The system flexible enough to assign different access privileges to each client according to the authorization level assigned to this client, in the same time it is pluggable in any existing system as it is implemented using pluggable filter.
The following table contains comparison between the proposed system, standard REST system and securing it using https protocol.
Non-secure REST Secured using https Proposed system
Authorization N N Y
Secure resources N Y Y
Secure URL/URI N N Y
Secure other protocols Y N Y

Table 1: Shows the comparison between using the proposed system, non-secure REST and securing it using https.


The system is composed of 3 main components: filter that control the access of these resources, access control list to apply all authorization roles and client data for authenticating the users, these authentication data need to store the 2 numbers shared between the client and the service provider, they are some sort password of the clients.
The system flexibility allows the usage of the implementation for selected resources only so it doesn’t impact the performance for the resources that do not need any security measures.

V. Conclusion
The proposed system cover some deficiencies in REST non-secure standard and provide some benefits include 1) securing the requested URL, 2) provide authorization through Access Control List (AOL) over resources, 3) secure the resources representation between the client and server being encrypted by the session key, 4) the system is implemented using pluggable filter so it can be easily switched on/off or added to any existing solution without much efforts.


References
[1] Fielding, Roy Thomas. “Architectural Styles and the Design of Network-based Software Architectures”. Doctoral dissertation, University of California, Irvine, 2000.
[2] Java Specification Request Web Site. http://jcp.org/en/jsr/detail?id=311 (Retrieved on 9th December, 2011).
[3] Paul Adamczyk, Patrick H. Smith, Ralph E. Johnson, Munawar Hafiz, REST and Web Services: In Theory and In Practice.
[4] Hypertext Transfer Protocol -- HTTP/1.0. http://tools.ietf.org/html/rfc1945 (Retrieved on 9th December, 2011).
[5] Hypertext Transfer Protocol -- HTTP/1.1. http://tools.ietf.org/html/rfc2616 (Retrieved on 9th December, 2011).
[6] Web Application Description Language (WADL). http://www.w3.org/Submission/wadl/ (Retrieved on 9th December, 2011).
[7] Martin E. Hellman et al. Cryptographic apparatus and method 1976.
[8] Why REST security doesn't exist - Chris Comerford and Pete Soderling. http://www.computerworld.com/s/article/print/9161699/Why_REST_security_doesn_t_exist?taxonomyName=Security&t
axonomyId=17 (Retrieved on 26th December, 2011).
[9] Securing REST Web Services With OAuth http://developers.sun.com/identity/reference/techart/restwebservices.html (Retrieved on 26th December, 2011).
[10] Eric Brachmann, Gero Dittmann ,Gero Dittmann. Simplified Authentication and Authorization for RESTful Services in Trusted Environments, August 2011.
[11] Eran Hammer-Lahav. The OAuth 1.0 protocol. RFC
5849, IETF, April 2010.
[12] Jacobs, I., Walsh, N., eds.: Architecture of the World Wide Web, Volume One, December 2004.

Cloud Computing Overview

Last Tuesday, 18th October , 2011 , I gave a session about Cloud Computing in HP ES Center in Cairo.. Here is a brief of what is included in that session...

Cloud Computing is The hottest buzzword in the IT world, It is information technology (IT) offered as a service; The selection of the term “Cloud“ is because over a decade, whenever people drew pictures of application architectures that involved the internet, they inevitably represented the internet with a cloud.

1) History :
One of the first movers in cloud computing was Salesforce.com, which in 1999 introduced the concept of delivering enterprise applications via a simple website.
Amazon was next by launching Amazon Web Service in 2002.
Then came Google Docs in 2006 which really brought cloud computing to the forefront of public consciousness.
2006 also saw the introduction of Amazon’s Elastic Compute cloud (EC2) as a commercial web service that allowed small companies and individuals to rent computers on which to run their own computer applications.
This was soon followed by an industry-wide collaboration in 2007 between Google, IBM and a number of universities across the United States.
Next came Eucalyptus in 2008, the first open source AWS API compatible platform for deploying private clouds, followed by OpenNebula, the first open source software for deploying private and hybrid clouds.
2009 saw Microsoft’s entry into cloud computing with the launch of Windows Azure in November.
Then, all the major players jumping on to cloud computing including Oracle , Dell, Fujitsu , Teradata , HP …etc…

2) Why Cloud Computing?
1.Eliminates the need for organizations to build and maintain expensive data centers.
2.Reduces the need for organizations to maintain a large IT staff.
3.Enables organizations to stand up new systems quickly and easily.
4.Provides elastic resources that allow applications to scale as needed in response to market demands.
5.Pay as-you-go rental model allows organizations to defer costs.
6.Increases business continuity by providing inexpensive disaster-recovery options.

3) Cloud main principles:
Whenever a system achieved these principles , you can named it as a cloud..
1.Pooled computing resources available to any subscribing users
2.Virtualized computing resources to maximize hardware utilization
3.Elastic scaling up or down according to need (Elasticity)
4.Automated creation of new virtual machines or deletion of existing ones
5.Resource usage billed only as used


4) Service & Deployment Models:



5) Virtual Private Cloud:
Secure VPN connection connecting private cloud with private assigned cloud subnet in the public cloud provider.


6) Cloud Pros and Cons:
Summarized in the following nice diagram:


7) Main Cloud Technologies:
1.A cloud needs servers on a network, and they need a home.
2.A cloud’s servers need to be virtualized.
3.A cloud needs an access API.
4.A cloud needs some storage.
5.Cloud application may need a database.
6.A cloud needs elasticity as a way to expand and contract applications.
7.A Cloud needs measure/monitor for payment.



8) Main Cloud Concerns:


9) Open Source Private Cloud:
Public cloud providers have primarily implemented their solutions with a combination of open source and homegrown software.
-Eucalyptus
-OpenNebula
Two open source initiatives, both offshoots of university research projects.
They provide a software capability for provisioning and managing a multi-user private cloud built on top of commodity hardware.
Both solutions compatible with the APIs provided by Amazon


10) Top Cloud Providers 2011:
1.Amazon (aws.amazon.com)
2.RackSpace (rackspacecloud.com)
3.Salesforce (salesforce.com)
4.Google (Google App Engine)
5.Microsoft (Windows Azure)
6.Joyent (joyent.com)
7.GoGrid (gogrid.com)
8.Terremark (terremark.com)
9.Savvis (savvis.com)
10.Verizon (verizonbusiness.com)

11) References:
“The Cloud at Your Service”, By Jothy Rosenberg & Arthur Mateos Cloud Computing
“Implementation, Management, and Security”, By John W. Rittinghouse & James F. Ransome, 2010
http://www.cpni.gov.uk/Docs/cloud-computing-briefing.pdf
http://www.cloudsecurityalliance.org/topthreats/csathreats.v1.0.pdf
http://www.cloudsecurityalliance.org/guidance/csaguide.v2.1.pdf
http://www.cpni.gov.uk/Docs/cloud-computing-briefing.pdf
http://www.hp.com/hpinfo/newsroom/press/2011/110125b.html
http://www.google.com/enterprise/cloud/appengine/pricing.html
http://h71028.www7.hp.com/enterprise/us/en/partners/vmware-hpvirtualsystem.html
http://www8.hp.com/us/en/software/software-product.html?compURI=tcm:245-936856
http://www.cloudtweaks.com/2011/02/a-history-of-cloud-computing/

Add Facebook Like Button In Java Website

The Facebook provides a lot of integration facilities so that your website become more social ..

One of these methods is to add the LIKE button (Share button is already depreciated now); you can also enable Facebook login/authentication, ...etc.

Here is the simple code you need to add to have this Like button working fine easily..



<iframe src="http://www.facebook.com/plugins/like.php?href=<%=request.getRequestURL() %>" scrolling="no" frameborder="0" style="border:none; width:450px; height:80px"></iframe>

That's it!



For more details about Facebook integration:
http://developers.facebook.com/docs/guides/web/

For more details about the Like button integration:
http://developers.facebook.com/docs/reference/plugins/like/

Simple Game Development Using Java

Balloon Game :

A simple Java game using the simple standard Java APIs without usage of any engine..

It is very small 10KB (+15 resources like images and sound) for educational purposes to show the Java developers the simple ways to use Java APIs in developing nice scenes without any complexity in the code.

The idea is controlling a balloon away from pins that can cause the balloon to explode.

In the next gaming post , i will show how to do complex Java games using 3D engines and/or Java3D.

The game scenes are entirely configurable , in this release i have made 11 scenes (from 0 to 10) but you can edit the property file to add million of scenes without the need to change any line of code in the game.

Download URL:

http://sourceforge.net/projects/balloongame/

Performance Tuning of ATG application

Before you start troubleshooting ATG application issue, you may do the following:
-Get the problem definition.
-Gather all possible information about it like affected transaction , when they detect it on load or with single page, in certain time or all over the day, in one managed server or all servers , when it start to occur , what changed before that time, any applied patch or new driver, ...etc...
-Get all possible tools and methods ready to use.
-Get the application code in-hands.
-Start building your investigation plan.
-Plan can have trial elements, gather information elements, possible fix elements and permanent fix/conclusion.
-If the issue is in the production env, try to replicate it in other env, so you can try fixing it without any possible business impact.

Here is a guide for your investigation and setup of the production environment for ATG application build from my experience plus ATG documentations :

1) ATG Application side Recommendations:

-Enabling liveconfig Settings:
When you’re ready to deploy your Nucleus-based application
in a production environment, enable the settings in the liveconfig configuration layer. This layer overrides many of the default configuration settings with values that are more appropriate for a deployed site. For example, the liveconfig configuration layer improves performance by reducing error checking and detection of modified properties files.

To enable liveconfig, you can use the –liveconfig argument for runAssembler
Or
or add the following line to the WEB-INF/ATGINF/
dynamo.env file in the atg_bootstrap.war module of your EAR file:
atg.dynamo.liveconfig=on

a) Disabling Checking for Changed Properties Files:
This property controls whether or how often ATG rereads .properties files or .java files the next time that an instance of a given component is created (non-global)
The default is 1000. This feature is useful during development, but we recommend disabling it once a site goes live for better performance. The value -1 disables the reloading of properties and .java files altogether.

b) Disable Performance Monitor:
/atg/dynamo/service/PerformanceMonitor) can be used to gather statistics about the performance of specific operations in ATG components.
You can disable the Performance Monitor by setting its mode property to 0:
mode=0
The Performance Monitor is disabled in the liveconfig configuration layer.

c) Adjusting the pageCheckSeconds Property:
ATG’s Page Processor compiles JHTML pages into .java files (JSP compilation is handled by your application server). The page processor, located at /atg/dynamo/servlet/pagecompile/PageProcessor, checks for new Java Server Pages that need to be compiled. You can improve performance by increasing the Page Processor’s pageCheckSeconds property. The page compile servlet uses this property value to determine whether to check for new Java Server Pages that need to be recompiled. If a request occurs within this time interval (measured in seconds) for the same page, ATG will not check the date on the file. This improves performance in serving pages.
A value of 0 causes ATG to check for new pages on each request. The default value is 1. The liveconfig value is 60.

-Fine-Tuning JDK Performance with HotSpot
Refer to Oracle Hot Sport performance tuning for more details.

-Configuring for Repositories:
a) Enable Caching:
Specify the correct values of cache according to your data size, example of this calculation:



b) Setting Cache Modes:
Select the proper cache mode
Remember that if you use locked mode caching, you must also enable lock manager components.

c) Populating Caches on Startup:
This benefit may come at the cost of slower startup times.
You can pre-populate caches in a SQL Repository by using tags in a repository definition file.

d) Configuring Repository Database Verification for Quicker Restarts:
By default, each SQL Repository component verifies each of the tables in its database on startup with a simple SQL query. These verification queries can slow the ATG startup routine.
you may wish to set the updateSchemaInfoCache property to true in your atg.adapter.gsa.GSARepository components, such as /atg/dynamo/service/jdbc/ProfileAdapterRepository.

e) Configure proper caching timings:
**item-cache-timeout :
This attribute defines how long (in milliseconds) a repository item can exist in the item cache without having been accessed before it needs to be reloaded from the database. Effectively, there is a "last touched" timestamp associated with each item cache entry; if the time since that item was last touched is greater than the item-cache-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

**item-expire-timeout :
This attribute defines how long (in milliseconds) a repository item can remain in the item cache before it needs to be reloaded from the database. Effectively, there is a "time loaded" timestamp associated with each item cache entry; if the time since that item was cached is greater than the item-expire-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

**query-expire-timeout :
This attribute is the same as the item-expire-timeout but for query cache entries. This attribute defines how long (in milliseconds) a query can remain in the query cache before it needs to be reloaded from the database. Effectively, there is a "time loaded" timestamp associated with each query cache entry; if the time since that entry was cached is greater than the query-expire-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

Note that the query-expire-timeout attribute only applies when you have query caching enabled.

-Setting Logging Levels :
If you want to disable logging entirely, or specify different logging levels, you can do that in the
GLOBAL.properties file. For example:
loggingError=true
loggingWarning=true
loggingInfo=true
loggingDebug=false

**Your application code must follow the standard in checking on the level before logging the log messages.

-Limiting Initial Services for Quicker Restarts
This is configured using initialServices property of the /atg/dynamo/Initial component.

-Disabling Document and Component Indexing
The ACC creates and maintains indexes of documents and components. For sites with large numbers of documents or components, indexing can take time and CPU resources. Once your site is deployed and relatively stable, you may want to limit or eliminate the indexing of documents or components.
The document and component indexes are maintained incrementally once built, and are rebuilt completely once a day at 1 a.m. by default. An index is rebuilt at startup only if it does not exist at all.
You can selectively exclude portions of the document tree from indexing by adding absolute pathname prefixes to the excludeDirectories property of the /atg/devtools/DocumentIndex component.
The same is true for component indexing, but the component is /atg/devtools/ComponentIndex instead. To improve performance on a live site, you can turn off all document and component indexing by setting the enabled property of the DocumentIndex and ComponentIndex components to false.

-Compress Content:
Compress pages (remove white-spaces) and static content (pictures, JS, CSS) which will speed up download time from the browser in your Web Server.
Compressing HTML/JavaScript/CSS/XML/JSON content can significantly reduce response times. GZIP reduces the size of responses by between 50% and 80%, depending on the type of content. Only turn on GZIP compression for text/html, text/plain, text/json, text/css, and application/x-javascript mime types.
To verify that gzip is being used, “Accept-Encoding: gzip,deflate” should show up in the request header and “Content-Encoding: gzip” should show up in the response header.

-Re-structure your page to have:
*Move .css to the top of all pages.
*Move script files to the button of all pages whenever possible.
*Move all embedded script into external files.

-Ajax Cache:
Cache your ajax response whenever possible to speed us the user experience whenever the results can be cached.

-Pre-Compiling JSPs:
Might slow deployment/server start-up but will speed up the 1st page request.

-Session Stickiness:
It’s important that session stickiness is working properly. Not having it working could result in sessions being continually restored after each request.

-Keep ATG Patched:
When possible, the latest version of ATG should be used. As cumulative patches are released, they should be applied.

-HTTP Connection Reuse:
Using the “Keep-Alive” header allows browsers to reuse the same TCP connection for multiple request/response pairs. Not re-establishing TCP connections for each HTTP request/response helps Reduce network traffic, Reduce the load on the SSL accelerator and Improve performance due to not having to setup and tear down TCP connections for each HTTP request/response.
A good Keep-Alive value for CSC is 300, which is 5 minutes.
Both Internet Explorer 6 and 7 forcibly terminate TCP connections after one minute, regardless of what Keep-Alive is set to. See http://support.microsoft.com/kb/813827 for a workaround.

2) Server\JVM\Operating system Configurations:
-Setup JDBC connections with number matching the expected concurrent users of the site.
-Setting the JTA timeout to 120 seconds whenever possible.
-Increase the max concurrent open files by the operating system.
-Compress the web application output.
-Configure Max Thread Stuck to a propert time to catch where the threads are usually stucked..

3) Using Performance measure tools:

A) ATG built-in Features:

1) Performance monitor:
*Adding PerformanceMonitor Methods to your Code
To enable the Performance Monitor to monitor a section of your Java code:
1. Import the atg.service.perfmonitor.* package.
2. Declare an opName parameter to label the section of the code. This parameter is displayed in the Performance Monitor page under the Operation heading.
3. (Optional) Declare a parameter name if you want to gather data on individual executions of an operation.
4. Call the startOperation method at the beginning of the operation whose performance you want to be able to measure.
5. Call the endOperation method at the end of the operation whose performance you want to be able to measure.
6. Optionally, call the cancelOperation method if an exception occurs. This causes the results of the current execution to be ignored.

These methods can be nested with different or the same opNames.
Example:
PerformanceMonitor.startOperation(opName, parameter);
try {
... code to actually render foo.jsp
} catch (Exception e) {
PerformanceMonitor.cancelOperation(opName, parameter);
exception = true;
} finally {
if (! exception)
PerformanceMonitor.endOperation(opName, parameter);
}

*Performance Monitor Modes:
You can set the Performance Monitor’s operating mode by setting the mode property of the component at /atg/dynamo/service/PerformanceMonitor:
disabled 0 (default)
normal 1
time 2
memory 3
You should use 2=Time to get accumulated results, also try to enable it after warm up the site to exclude extreme reading for loading caches ,...etc.

*View the Results:
You can view the information collected by the Performance Monitor on the Performance Monitor’s page
of the Dynamo Administration UI at:
http://hostname:port/dyn/admin/atg/dynamo/admin/en/performance-monitor.jhtml

2) Using the VMSystem Component :
The ATG component located at /VMSystem provides a way for you to access the Java memory manager.
You can monitor the status of the Virtual Machine and call methods on it. An interface to the VMSystem component is included in the Dynamo Administration UI at:
http://hostname:port/dyn/admin/nucleus/VMSystem/
From this page, you can conduct the following VM Operations:
• Perform garbage collection
• Run finalizations
• Show memory information
• List system properties
• List thread groups
• List threads
• Stop the VM

3) Sampler:
When testing your site, it is useful to automatically sample performance to understand throughput as a function of load. ATG includes a Sampler component at /atg/dynamo/service/Sampler.
Starting the Sampler You can start the Sampler component by opening it in the ACC and clicking the Start button.
You can also start the Sampler component from the Dynamo Administration UI by requesting this URL:
http://hostname:port/dyn/admin/nucleus/atg/dynamo/service/Sampler
The first time you request this page, ATG instantiates the Sampler component, which begins recording statistics.
You can configure ATG to start the Sampler whenever ATG starts by adding the Sampler to the initialServices property of the /atg/dynamo/service/Initial component:

The Sampler outputs information to the file /home/logs/samples.log. For each system variable that it samples, it records the following information in the log file:
• the current value
• the difference between the current value and the value recorded the last minute
• the rate of change of the value
You can adjust values recorded by the Sampler, but the default set is comprehensive in monitoring ATG request handling performance.

B) Log Files:

* Access Logs:
You may enabled access logs on your application sercer or web server to ensure that the time is really consumed inside your application not in network traffic (download time)..

* Application Logs:
Application logs could point to system calls or external systems timeout or DB issues , exceptions , .... alot of useful information could be detected from the application logs.
You might see a “server not responding” message or an OutOfMemory error

C) Thread Dump:
Thread dumps can be useful to see where these threads are
waiting. If there are too many threads waiting, your site’s performance may be impaired by thread context switching. You might see throughput decrease as load increases if your server were spending too much time context-switching between requests. Check the percentage of System CPU time consumed by your JVM. If this is more than 10% to 20%, this is potentially a problem. Thread context switching also depends
in part on how your JVM schedules threads with different priorities

Thread dumps can be taken from Admin console of the command line.
Thread dumps could also point easily to deadlocks and infinite loops and other issues related to bad buggy code.

D) Garbage Collection:
Check the JVM parameters that affect Garbage Collection including gc policy, try to set max and min heap size the same value.
Check the garbage collections Logs; typically when you see spikes or abnormal behaviors, the most important is the full garbage collection runs.

*Phases that stop the threads with the parallel garbage collection algorithm:
tail -f | grep 'Full'
*Phases that stop the threads with the Concurrent Mark Sweep algorithm:
tail -f | grep -E '(CMS-initial-mark)|(Rescan )|(concurrent mode
failure)|(Trying a full collection)|(promotion failed)|(full)|(Full)'

If excessive pausing is noticed, one of several things could be wrong:
- JVM arguments may need to be tuned
- There may be a memory leak
- Repository/droplet/atg.service.cache.Cache caches may be over-utilized
- Load balancing could be not working properly, which would result in more sessions than normal hitting one instance.

While thread pauses are a normal part of garbage collection, excessive pauses must be minimized.

E) Cron Jobs:
Check the running cron-jobs on the server and try to only setup them to run during non-load hours of the servers.

F) Profiling Tools:
-Netbeans Profiler
-JProfiler
-Eclipse TPTP
Any other profiling tools that give you detailed information about consumed in any operation and memory tracing ,...etc.

G) Load testing Tools:
*) URLHammer (ATG load tool):
To run the URLHammer program:
1. Set your CLASSPATH to include the directory /DAS/lib/classes.jar.
2. Run the following command:
java atg.core.net.URLHammer [arguments]
For example:
java atg.core.net.URLHammer http://examplehost:8840/ 5 10 -cookies
This creates five different threads, each of which represents a separate session that requests the specified
URL 10 times (50 requests total).

You can also run a script by editing the format yourself or by using RecordingServlet to create it..

*) Apache JMeter (open source)
*) HP Load Runner (commercial)

H) Client Side Performance Tools:
-Firebug (FF plugin)
-Fiddler (Standalone or plugin)
-DynaTrace (Standalone or plugin)
The most important is to identify if a certain resources (esp outside your domain) is taking much of the time.
Invalid configuration also could be detected , pointing to another env.
JS performance might be a reason for end-user bad performance.

I) Operating system performance Tools:
Monitoring System Utilization Use a program like top (on Solaris), the Windows Performance Monitor, or a more sophisticated tool to keep track of information like:
• CPU utilization
• paging activity
• disk I/O utilization
• network I/O utilization
*If you are getiing Can't create native thread exception , you should know that your main system memory is low , as each native thread reserve around 1 MB of memory for its stack trace.
*You can detect a file descriptor leak in two different ways:
• You may notice a lot of IOExceptions with the message “Too many open files.”
• During load testing, you periodically run a profiling script, such as lsof (on UNIX), and you notice that the list of file descriptors grows continually.

J) DB performance tuning:
- Take DB snapshots and analysis them.
- Take the most consuming sql and do execution plan for them, possibly you could find a missing index.
- You may need a DBA to analyze possible DB issue.
- Enable JDBC logging and retrieve the SQL queries and try to optimize them outside the application (debug level need to be set to 15, you can also retrieve this queries from DB monitor tools)

K) Adjusting the FileCache Size :
ATG’s servlet pipeline includes servlets that are used for JHTML pages, and which use a FileCache component to store files that ATG has read from disk, so that subsequent accesses for those files can be delivered directly from memory instead of being read from disk. Using the FileCache component improves performance by reducing disk accesses. For maximum performance, you want the FileCache to be large enough to hold all the files that ATG serves frequently.
Set the totalSize property of this component at:
/atg/dynamo/servlet/pipeline/FileCache
to an appropriate value, measured in bytes, such as the following:
# size in bytes (2 million bytes)
totalSize=2000000

One approach in sizing the FileCache is to batch compile the entire document root and set the file cache to the resulting size. Make sure, however, that you account for the size of your FileCache when
you set the size of your JVM. You can preload the FileCache by creating a script that accesses every page on your site and running the script on startup.
You can view statistics on how the file cache is used, as well as the contents of the FileCache in the Dynamo Administration page at
hostname:port/dyn/admin/nucleus/atg/dynamo/servlet/pipeline/FileCache.

L) Code optimization:
You may scan the code in the high transaction scenarios, with special care to the pipelines and component scopes to identify possible performance issues..
You may use tools for code optimizations.
Some finding may need changes to component scope (the global is best performing being loaded only once, while the request scoped is initialized per each request, so you need to minimize it)
Avoid resolving the components from the code, instead using ATG property files to inject them , in case you do not have that ability like derived properties , you may have static method to get a reference to them (if global components).
Follow the code standards and best practices..
Once possible reason for application bad performance is not following the logging best practices by checking the log level before logging the message.

M) Use Cache Droplet in your JSP pages:
Cache Droplet caches content that changes infrequently used especially if it includes a lot of processing or DB interactions (Component /atg/dynamo/droplet/Cache)
**Required Input Parameters key
Lets you have more than one view of content based on a value that uniquely defines the view of the content. For example, if content is displayed one way for members and another for non-members, you can pass in the value of the member trait as the key parameter.
**Optional Input Parameters
-hasNoURLs
Determines how cached URLs are rendered for future requests. By setting hasNoURLs to false, you specify that subsequent requests for the cached content causes URLs to be rewritten on the fly, assuming URL Rewriting is enabled. A setting of false for hasNoURLs causes URLs to be saved and rendered exactly as they are currently (without session or request IDs) regardless of whether URL rewriting is enabled.

-cacheCheckSeconds
The interval after content is cached until the cached is regenerated. If omitted, the interval is set from the defaultCacheCheckSeconds property in the Cache servlet bean’s properties file.

**Open Parameters output
The code enclosed by the output open parameter is cached.

**Clearing the cache:
You can determine how often data is flushed for a given Cache instance on a JSP or for all instances of Cache. To remove cached content associated with a particular instance of Cache, set the cacheCheckSession input parameter in the Cache instance to the frequency by which associated data should be expired. If you omit this parameter, the Cache.defaultCacheCheckSeconds property is used (default value is 60 seconds) .
The Cache.purgeCacheSeconds property determines how often content cached by any Cache servlet bean is flushed. The default is 21600 seconds (6 hours). Cache purging also occurs when a JSP is removed or recompiled.

N) Hardware limited capacity:
This case for development environment where you may consider moving one application outside this box, like the DB , Merch , ...etc..
Also you can shutdown the Merch if you do not need it up and running all the time.
Another trick -strictly for Dev box- is decreasing session timeout into 5 minutes or decrease the thread reserved memory usage.


O) ATG recommended Check List:
The following checklist can help you identify the most common sources of performance problems:
• Have you properly configured memory for your Java Virtual Machines? Have you set your -Xms and -Xmx arguments the same? Do all ATG heap sizes fall within the limits of physical memory?
• Has one or more servers stopped responding? There could be a number of causes, including a Java deadlock.
• Are you seeing many IOExceptions with the message “Too many open files”? You may have a file descriptor leak.
• At maximum throughput, look at the CPU utilization, database CPU utilization, I/O activity, and paging activity.
• If CPU utilization is low, then you may have an I/O or database bottleneck.
• If CPU utilization is high, then the bottleneck is most likely in the application code. Use a performance profiling tool to try to locate bottlenecks in the code. Review your code to make sure it uses good Java programming practices.
• If paging is occurring, adjust the memory allocated to your Java Virtual Machines.
• Look at the I/O and CPU utilization of the database. If utilization is high, database activity is probably slowing down the application.
• Are you receiving page compilation errors? You may not have enough swap space for page compilation.


Reference: ATG Platform documentation set : Version 9.1 - 7/31/09 and others..
For More information, refer to Java EE 7 performance tuning and optimization book: The book is published by Packt Publishing: http://www.packtpub.com/java-ee-7-performance-tuning-and-optimization/book

ATG Made Easy - part 6

ATG Commerce
2 versions: ATG Consumer Commerce is used for developing standard business-to-consumer (B2C) online stores. ATG Business Commerce is used for sites oriented more toward business-to-business (B2B) uses.

1) Product Catalog & Custom Catalog:
The product catalog is a collection of repository items (categories, products, media, etc.) that provides the organizational framework for your commerce site. ATG Commerce includes a standard catalog implementation, based on the ATG SQL Repository, that you can use or extend as necessary.

The structure is built on Catalog --> Category --> Product --> Sku
Where you can modify the features of each level.
The most important is to have the parent flag in the top categories
Each level has parent/child attributes, eg. parentCategories, childSku's for product level.
You can have a linked template to display the element using it.
Some dropletes are dedicated on loading these elements: ItemLookupDroplet (generic) versus CategoryLookupDroplet, ProductLookupDroplet and SKULookupDroplet.
They take id as input and product the item/elements as output.

MediaLookupDroplet for Media , 2 types exist: Media Internal and External (internally referenced versus URL/File)
The data property is either binary (for media-internal-binary items) or text (for media-internal-text items)

ATG catalogs use several batch and dynamic services to perform catalog updates and to verify catalog relationships. These services are collectively referred to as the Catalog Maintenance System (CMS). The CMS updates property values that enable navigation and hierarchical search. It also verifies the relationships of catalog repository items and properties.
**Batch Services : as
• AncestorGeneratorService
• CatalogVerificationService
• CatalogUpdateService
• CatalogMaintenanceService
Each service creates a global lock at the start of execution, using the global lock management, and releases it upon completion. This prevents services that use the same lock name from executing simultaneously on the same server, or other servers in the cluster.
**Dynamic Services : as
• CatalogChangesListener
• CatalogCompletionService
• StandardCatalogCompletionService

Note that some of these services are available only for custom catalogs; others for both standard and
custom catalogs.

2) Order Management:
A session scoped component which is ShoppingCart (atg/commerce/ShoppingCart) manage the current order (+last order and saved orders), order stored in memory and persisted frequently..
You can configure and customize it using orderrepositiory.xml (where you can define its components, cache size and lock mode).
Once Sku is added to the order-->it become CommerceItem that is related to ShoppingGroup that is related to ShippingAddress and ShippingMethod.
Once order is submitted , it become in incomplete status ---> further processing --> fulfilled.
In case of B2B, order is hold for the needed approvals.
Once good feature exist in ATG Commerce, is the ability to have scheduled order.

Payment methods include: Credit card, Store credit, gift certificate, invoice request (in B2B)

CartModifierFormHandler: Request scope
A form handler to handle adding items to order, remove items, change quantity, move the order forward (payment, shipping info),

StoreCartFormHandler:
Handles continue shopping, expressCheckout, update, ....etc...

*Business Layer:
-CommerceItemManager:
Create, add to order, remove from order..
Once sku is added to the order : sku id--> catalog ref Id in CommerceItem object, quantity , priceInfo , paymentGroup and shippingGroup are taken as well.

-OrderManager:
load/save, create , allocate order to payment group.
These operations are handled by pipelines as mentioned before when we discuss the pipelines.
-OrderTools:
Provides low level raw operations..
-PurchaseProcessHelper:
Centralized functionality for purchase operations.

*Best Practise for Order Updating:
1.Acquire lock-manager write lock on profile id from the /atg/commerce/order/LocalLockManager
2.Begin Transaction
3.Synchronize on the Order object.
4.Modify Order
5.Call ((OrderImpl) pOrder).updateVersion();
6.Call OrderManager.updateOrder()
7.Release Order synchronization
8.End Transaction
9.Release lock-manager write lock on profile id from the /atg/commerce/order/LocalLockManager


3) Purchasing and Fulfillment Services :
ATG Commerce provides tools to handle pre-checkout order-processing tasks such as adding items to a shopping cart, ensuring items are shipped by the customer’s preferred method, and validating credit card information. The system is designed for flexibility and easy customization; you can create sites that support multiple shopping carts for a single user, multiple payment methods and shipping addresses.
You can integrate with third-party authorization and settlement tools such as Payflow Pro, CyberSource, and TAXWARE.

4) Inventory Management :
The inventory framework facilitates inventory querying and inventory management for your site.
InventoryManager
The InventoryManager is a public interface that contains all of the Inventory system functionality. Each method described below returns an integer status code. All successful return codes should be greater than or equal to zero, all failure codes should be less than zero. By default, the codes are:
INVENTORY_STATUS_SUCCEED=0 There was no problem performing the operation.
INVENTORY_STATUS_FAIL=-1 There was an unknown/generic problem performing the operation.
INVENTORY_STATUS_INSUFFICIENT_SUPPLY = -2 The operation couldn’t be completed because there were not enough of the item in the inventory system.
INVENTORY_STATUS_ITEM_NOT_FOUND = -3 The operation could not be completed
because a specified item could not be found in the inventory system.

ATG Commerce includes the following implementations of the InventoryManager out of the box.
• AbstractInventoryManagerImpl
• NoInventoryManager
• RepositoryInventoryManager
• CachingInventoryManager
• LocalizingInventoryManager

Preventing Inventory Deadlocks
InventoryManager includes the acquireInventoryLocks and releaseInventoryLocks methods.
acquireInventoryLocks acquires locks for the inventory items that apply to the given IDs.
releaseInventoryLocks releases locks for the inventory items that apply to the given IDs.

5) Pricing Services :
ATG Commerce pricing services revolve around pricing engines and pricing calculators.
**The pricing engine determines the correct pricing model for an order, individual item, shipping charge, or tax, based on a customer’s profile.
**The pricing calculator performs the actual price calculation based on information
from the pricing engine.

Pricing engines are responsible for three tasks:
• Retrieving any promotions that are available to the site visitor.
• Determining which calculators generate the price.
• Invoking the calculators in the correct order.

Pricing calculators are responsible for the following:
• Looking up the price in the catalog by priceList.
• Invoking a qualifier service that identifies the objects to discount.
• Using information they receive from the engines and from the qualifier service to
perform the actual process of determining prices.

By default, ATG Commerce can perform dynamic pricing for the following types of pricing object:
• Items. Each item has a list price that can be specified in the listPrice property of the Product Catalog repository.(Note that an “item” is a CommerceItem, which represents a quantity of a SKU or a product).
• Orders.
• Shipping price.
• Tax.

Terminology:
-Qualifier : This is a service that interprets a PMDL rule and decides what, if
anything, may be discounted. The term qualifier also refers to the first part of a PMDL rule. This rule defines when something can receive a discount.
- Target : The second part of a PMDL rule is called the target. This rule
defines which part of an order receives a discount.

4 Types of PriceInfo objects:
OrderPriceInfo,ItemPriceInfo, ShippingPriceInfo, and TaxPriceInfo

How this works?
1-Invokation to price engine from business-layer logic, such as a PriceItem servlet
bean in a page or from the ATG Commerce PricingTools class.
2-Pricing engine applies its configured precalculators. A precalculator modifies a
price without using any associated promotions.
3-Pricing engine accesses the current customer’s profile and retrieves any
promotions listed in the activePromotions property of the profile
4-Pricing engine builds a list of global promotions and concatenate them.
5-Pricing engine applies promotions by priority (each promotion has pricingCalculatorService property that specifies the calculator that the system must use to apply it)
6-The pricing engine applies its configured PostCalculators.
7-The pricing engine modifies the PriceInfo object of the object being discounted.

-When building a product catalog, you must decide whether your site requires dynamic product pricing and, if so, how granular you need it to be. Using dynamic pricing on a product page can cause a significant decrease in performance compared to using static pricing..

-With static pricing, each item in the catalog has a list price stored in the listPrice property of the catalog repository.
Volume Pricing: Bulk 100 at 10, 200 at 9, ...etc..
or Tiered: 1st 100->10 , 2nd 100 -->9 , ...
Static uses 2 droplets: PriceDroplet and PriceRangeDroplet.

**PriceEngine:
atg.commerce.pricing.PricingEngine is the main interface for interacting with the
atg.commerce.pricing package. Extensions of this interface describe objects that calculate a price for a specific class of object. For example, OrderPricingEngine extends PricingEngine and calculates prices for orders passed to it.

All PricingEngine implementations process promotions. The PricingEngine interface itself contains only one method, getPricingModels, which extracts a collection of promotions from an input profile. (Price Model can be customized by priceModel.xml).
The PricingEngine interface itself does not describe any functionality other than the promotion extraction API because of the wide range of information that different PricingEngine implementations might require to calculate a price for their specific class of object. For example, the ItemPricingEngine implementation needs one set of input parameters, while the OrderPricingEngine needs a different set.

The individual extensions of the PricingEngine interface contain the API methods for generating a given type of price. There is a Java object type for each type of price that is generated. For example,atg.commerce.pricing.OrderPricingEngine inherits the promotion extraction API from PricingEngine and defines one new method, priceOrder, to generate a price for an order in a given context.

ATG Commerce provides the following four extensions of the main PricingEngine interface:
• atg.commerce.pricing.ItemPricingEngine
Provides a price for atg.commerce.order.CommerceItem objects.
• atg.commerce.pricing.OrderPricingEngine
Provides a price for atg.commerce.order.Order objects.
• atg.commerce.pricing.ShippingPricingEngine
Provides a price for atg.commerce.order.ShippingGroup objects.
• atg.commerce.pricing.TaxPricingEngine
Determines tax for atg.commerce.order.Order objects.

PricingTools Class
The atg.commerce.pricing.PricingTools class performs a variety of pricing functions for different types of pricing engines. It also has a number of static, currency-related methods for use by all pricing engines.
The PricingTools class is the main way that business-layer logic interacts with the pricing engines and the other classes in the atg.commerce.pricing package.

The properties of PricingTools are as follows:
• itemPricingEngine: The pricing engine that calculates prices for items, both
individually and in groups. An item is identified as a set quantity of a SKU or product.
• orderPricingEngine: The pricing engine that calculates prices for orders. Typically,
the price is just the sum of the prices of the items in the order. However, the order
might be discounted separately from the constituent items.
• shippingPricingEngine: The pricing engine that calculates prices for shipping
groups. An order contains one or more shipping groups when the contents of the
order require shipping for delivery. An order has no shipping groups when it is
delivered online, and shipping is therefore not calculated for this type of order.
• taxPricingEngine: The pricing engine that calculates tax for orders. Tax is calculated
on the order total.
• roundingDecimalPlaces: Specifies the number of decimal places to which the an
input price is rounded . This property is used by the round, roundDown and
needsRounding methods.

Important Methods:
priceEachItem, priceItem, priceItemsForOrderTotal, priceOrderForOrderTotal, priceOrderTotal, priceShippingForOrderTotal, priceTaxForOrderTotal, needsRounding, round.

The Pricing Servlet Beans : You can insert on site pages as required and use to
perform dynamic pricing.
• AvailableShippingMethods Servlet Bean
• ItemPricingDroplet Servlet Bean
• PriceEachItem Servlet Bean
• PriceItem Servlet Bean
• PriceDroplet Servlet Bean
• ComplexPriceDroplet Servlet Bean

*Price Lists:
Price Lists allow you to target a specific set of prices to a specific group of customers.
The PriceListManager class maintains the price lists. A price may be retrieved from the PriceListManager from a given price list by product, by SKU, or by a product/SKU pair.
The most important method in PriceListManager is getPrice. This is used during pricing of an order to get the correct price for a given product/SKU pair.

It is configured using priceList.xml

6) Targeted Promotions :
Business managers can use ATG Commerce promotions to highlight products and offer discounts as a way of encouraging customers to make purchases.

Promotions 3 types: ItemDiscount, OrderDiscount and ShippingDiscount.

Pricing Model Description Language (PMDL): rule describing the conditions under which this promotion should take effect. The rule is created in the ATG Control Center using the interface designed for promotion creation.

7) Order Management Web Services :
All order management web services are included in commerceWebServices.ear in the
orderManagement.war web application.

Reference: ATG Platform documentation set : Version 9.1 - 7/31/09