Saturday, December 31, 2011

Using Diffie–Hellman Key Exchange to Secure RESTful WS

Osama M. Oransa
Arab Academy for Science, Technology & Maritime Transport


Abstract
Representational State Transfer (REST) [1] is an architectural style for building large-scale distributed information networks, the most famous of these being the World Wide Web [12]. RESTful web services have an issue with its security model, lacking a standard way to secure it, in the same time its security relies in the underlying protocol. [8]

Existing solution to secure it, is to use https instead of http, the issue with that approach is that , this is a transport layer protocol so it doesn’t provide authentication or authorization yet it secure the end to end channel between both parties.

In the proposed solution, Diffie–Hellman key exchange is used to generate a session key to be used within the interactions to secure the requested resources and encrypt the resource representation or parts of it; in this solution a pluggable filter is introduced with the advantages of avoid any change in the existing code and secure only the needed resources.

The suggested solution protect the resources and provide both authentication and authorization, this would fix the security issues related to RESTful web services but results in stateful filter that contradict with the basic concepts of this technology.

Keywords
RESTful Web Services; Diffie–Hellman Key Exchange; Web Services Security.



I. Introduction
RESTful web services emerged as a scalable way of doing web services yet there is not mature as the traditional SOAP-based web services that has a lot of security standards including WS-SE??.

Some suggested solutions to secure the RESTful web services includes using https instead of http, the issue with that approach is that , this is a transport layer protocol so it doesn’t provide authentication or authorization yet it secure the end to end channel between both parties, anther way is to send authentication data with each request, this could provide authentication , but subjected to other attacks in case as man-in-the-middle attack, another way is to send a token granted from token generating service like OAuth[11] this have the same previous solution [9] most successful if used in trusted environment like organization intranet [10].
In the proposed solution, Diffie–Hellman key exchange is used to generate a session key to be used within the interactions to secure the requested resources and encrypt the resource representation or parts of it; in this solution a pluggable filter is introduced with the advantages of avoid any change in the existing code and secure only the needed resources.


II. Background
REST refers to Representational state transfer, a terminology first mentioned by Fielding, Roy Thomas in his Doctoral dissertation “Architectural Styles and the Design of Network-based Software Architectures” in the year of 2000. [1]

The concept is known since the evolution of the internet but it was first time to be described and this make sense as Fielding was one of the authors of http protocol specifications (1.0 and 1.1)[4][5], the 1st interest in this architecture appears once the programming languages support the REST feature; Java in 2007/2008 [2]

In spite the wide adaptation of RESTful web services in the public exposed services; it is still has some areas that need to be enhanced in compare to SOAP-based web services where there are a lot of existing standards referred as WS-* standards (e.g. WS-Addressing, WS-Security); one of these areas is the web service security where it lack any standard way to implement it and actually the only existing standard way is to use https protocol as a transport layer security protocol.

For message-level security, it’s using HTTPS seems enough, but for other capabilities such as signatures, encryption, or federation cannot be supplied by HTTP alone. [3]
One existing solution is to have authentication token by a secure token service then use the token to access this RESTful web services. [3] The standard description of the RESTful web services is Web Application Description Language (WADL). [6]

Diffie–Hellman key exchange method - first published by Whitfield Diffie and Martin Hellman in 1976 - allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure communications channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher. [7]

III. Proposed Model
The proposed solution here utilize the Diffie–Hellman key exchange to generate a session key using the normal http protocol without the need for any secure tunneling so the client and the server can both have the same key and use it to encrypt the parts the need to be secure in the REST calls.
For example the resource URL can be encrypted , and only the server can identify it using the session key, also the returned resource could be also encrypted (partially or completely) so only the client requested this resource can decrypt it using the session key.
This is implemented using intercepting filter that authorize the client and validate the resource access rights, the main advantage of being implemented as a filter is that it can be configured per the needed resources so you can still provide some without security, also no need to change the RESTful WS implementation as it would be just another layer over it so its pluggable, and you don’t need the code to be repeated being centralized location for these requests.
The server will invalidate the key after a configurable time (session time) in that case the client is required to generate a new session key to interact, this filter here represents the only part of the system that behave as stateful not stateless as the RESTful web services behavior.



Figure 1: Represents the different component in the system and different message types between the client and our proposed RESTful System.

As shown in the above diagram the system has client data represents the needed information which are the identification and 2 selected values to generate the session key, another component to authorize each user to different resources, describes which component could be interacted by the client and to which level (read, create, update and delete).
The client can interact in 2 steps 1st to get a session key then to manipulate the resources (step 1 and 2 in the diagram), of course the session key will expire after certain period of time so the client call need to be wrapped to get a new session key if the call return session key expired exception or to have some sort of scheduler run to get a new session key, the session key here represent a resource in the system like any other resources so the user need to interact with it on this concept; it is a good way to increase the security over the connection as without the need of https the channel is dynamically secure, yet the user can choose if the resource is needed encrypted or not according to the needs plus the ACL that determine additional attribute if this resource is allowed to be interacted non-encrypted.

IV. Results
The system is used to control integration layer between the 2 tiers , the system provide a good way to secure RESTful web services without the need to use https, the security is dynamically obtained using a session key generated by Diffie–Hellman key exchange method, also the system provide a way to authorize the access on different resources.
The system flexible enough to assign different access privileges to each client according to the authorization level assigned to this client, in the same time it is pluggable in any existing system as it is implemented using pluggable filter.
The following table contains comparison between the proposed system, standard REST system and securing it using https protocol.
Non-secure REST Secured using https Proposed system
Authorization N N Y
Secure resources N Y Y
Secure URL/URI N N Y
Secure other protocols Y N Y

Table 1: Shows the comparison between using the proposed system, non-secure REST and securing it using https.


The system is composed of 3 main components: filter that control the access of these resources, access control list to apply all authorization roles and client data for authenticating the users, these authentication data need to store the 2 numbers shared between the client and the service provider, they are some sort password of the clients.
The system flexibility allows the usage of the implementation for selected resources only so it doesn’t impact the performance for the resources that do not need any security measures.

V. Conclusion
The proposed system cover some deficiencies in REST non-secure standard and provide some benefits include 1) securing the requested URL, 2) provide authorization through Access Control List (AOL) over resources, 3) secure the resources representation between the client and server being encrypted by the session key, 4) the system is implemented using pluggable filter so it can be easily switched on/off or added to any existing solution without much efforts.


References
[1] Fielding, Roy Thomas. “Architectural Styles and the Design of Network-based Software Architectures”. Doctoral dissertation, University of California, Irvine, 2000.
[2] Java Specification Request Web Site. http://jcp.org/en/jsr/detail?id=311 (Retrieved on 9th December, 2011).
[3] Paul Adamczyk, Patrick H. Smith, Ralph E. Johnson, Munawar Hafiz, REST and Web Services: In Theory and In Practice.
[4] Hypertext Transfer Protocol -- HTTP/1.0. http://tools.ietf.org/html/rfc1945 (Retrieved on 9th December, 2011).
[5] Hypertext Transfer Protocol -- HTTP/1.1. http://tools.ietf.org/html/rfc2616 (Retrieved on 9th December, 2011).
[6] Web Application Description Language (WADL). http://www.w3.org/Submission/wadl/ (Retrieved on 9th December, 2011).
[7] Martin E. Hellman et al. Cryptographic apparatus and method 1976.
[8] Why REST security doesn't exist - Chris Comerford and Pete Soderling. http://www.computerworld.com/s/article/print/9161699/Why_REST_security_doesn_t_exist?taxonomyName=Security&t
axonomyId=17 (Retrieved on 26th December, 2011).
[9] Securing REST Web Services With OAuth http://developers.sun.com/identity/reference/techart/restwebservices.html (Retrieved on 26th December, 2011).
[10] Eric Brachmann, Gero Dittmann ,Gero Dittmann. Simplified Authentication and Authorization for RESTful Services in Trusted Environments, August 2011.
[11] Eran Hammer-Lahav. The OAuth 1.0 protocol. RFC
5849, IETF, April 2010.
[12] Jacobs, I., Walsh, N., eds.: Architecture of the World Wide Web, Volume One, December 2004.

Sunday, October 23, 2011

Cloud Computing Overview



Last Tuesday, 18th October , 2011 , I gave a session about Cloud Computing in HP ES Center in Cairo.. Here is a brief of what is included in that session...

Cloud Computing is The hottest buzzword in the IT world, It is information technology (IT) offered as a service; The selection of the term “Cloud“ is because over a decade, whenever people drew pictures of application architectures that involved the internet, they inevitably represented the internet with a cloud.

1) History :
One of the first movers in cloud computing was Salesforce.com, which in 1999 introduced the concept of delivering enterprise applications via a simple website.
Amazon was next by launching Amazon Web Service in 2002.
Then came Google Docs in 2006 which really brought cloud computing to the forefront of public consciousness.
2006 also saw the introduction of Amazon’s Elastic Compute cloud (EC2) as a commercial web service that allowed small companies and individuals to rent computers on which to run their own computer applications.
This was soon followed by an industry-wide collaboration in 2007 between Google, IBM and a number of universities across the United States.
Next came Eucalyptus in 2008, the first open source AWS API compatible platform for deploying private clouds, followed by OpenNebula, the first open source software for deploying private and hybrid clouds.
2009 saw Microsoft’s entry into cloud computing with the launch of Windows Azure in November.
Then, all the major players jumping on to cloud computing including Oracle , Dell, Fujitsu , Teradata , HP …etc…

2) Why Cloud Computing?
1.Eliminates the need for organizations to build and maintain expensive data centers.
2.Reduces the need for organizations to maintain a large IT staff.
3.Enables organizations to stand up new systems quickly and easily.
4.Provides elastic resources that allow applications to scale as needed in response to market demands.
5.Pay as-you-go rental model allows organizations to defer costs.
6.Increases business continuity by providing inexpensive disaster-recovery options.

3) Cloud main principles:
Whenever a system achieved these principles , you can named it as a cloud..
1.Pooled computing resources available to any subscribing users
2.Virtualized computing resources to maximize hardware utilization
3.Elastic scaling up or down according to need (Elasticity)
4.Automated creation of new virtual machines or deletion of existing ones
5.Resource usage billed only as used


4) Service & Deployment Models:



5) Virtual Private Cloud:
Secure VPN connection connecting private cloud with private assigned cloud subnet in the public cloud provider.


6) Cloud Pros and Cons:
Summarized in the following nice diagram:


7) Main Cloud Technologies:
1.A cloud needs servers on a network, and they need a home.
2.A cloud’s servers need to be virtualized.
3.A cloud needs an access API.
4.A cloud needs some storage.
5.Cloud application may need a database.
6.A cloud needs elasticity as a way to expand and contract applications.
7.A Cloud needs measure/monitor for payment.



8) Main Cloud Concerns:


9) Open Source Private Cloud:
Public cloud providers have primarily implemented their solutions with a combination of open source and homegrown software.
-Eucalyptus
-OpenNebula
Two open source initiatives, both offshoots of university research projects.
They provide a software capability for provisioning and managing a multi-user private cloud built on top of commodity hardware.
Both solutions compatible with the APIs provided by Amazon


10) Top Cloud Providers 2011:
1.Amazon (aws.amazon.com)
2.RackSpace (rackspacecloud.com)
3.Salesforce (salesforce.com)
4.Google (Google App Engine)
5.Microsoft (Windows Azure)
6.Joyent (joyent.com)
7.GoGrid (gogrid.com)
8.Terremark (terremark.com)
9.Savvis (savvis.com)
10.Verizon (verizonbusiness.com)

11) References:
“The Cloud at Your Service”, By Jothy Rosenberg & Arthur Mateos Cloud Computing
“Implementation, Management, and Security”, By John W. Rittinghouse & James F. Ransome, 2010
http://www.cpni.gov.uk/Docs/cloud-computing-briefing.pdf
http://www.cloudsecurityalliance.org/topthreats/csathreats.v1.0.pdf
http://www.cloudsecurityalliance.org/guidance/csaguide.v2.1.pdf
http://www.cpni.gov.uk/Docs/cloud-computing-briefing.pdf
http://www.hp.com/hpinfo/newsroom/press/2011/110125b.html
http://www.google.com/enterprise/cloud/appengine/pricing.html
http://h71028.www7.hp.com/enterprise/us/en/partners/vmware-hpvirtualsystem.html
http://www8.hp.com/us/en/software/software-product.html?compURI=tcm:245-936856
http://www.cloudtweaks.com/2011/02/a-history-of-cloud-computing/

Sunday, October 9, 2011

Add Facebook Like Button In Java Website

The Facebook provides a lot of integration facilities so that your website become more social ..

One of these methods is to add the LIKE button (Share button is already depreciated now); you can also enable Facebook login/authentication, ...etc.

Here is the simple code you need to add to have this Like button working fine easily..



<iframe src="http://www.facebook.com/plugins/like.php?href=<%=request.getRequestURL() %>" scrolling="no" frameborder="0" style="border:none; width:450px; height:80px"></iframe>

That's it!



For more details about Facebook integration:
http://developers.facebook.com/docs/guides/web/

For more details about the Like button integration:
http://developers.facebook.com/docs/reference/plugins/like/

Saturday, September 24, 2011

Simple Game Development Using Java




Balloon Game :

A simple Java game using the simple standard Java APIs without usage of any engine..

It is very small 10KB (+15 resources like images and sound) for educational purposes to show the Java developers the simple ways to use Java APIs in developing nice scenes without any complexity in the code.

The idea is controlling a balloon away from pins that can cause the balloon to explode.

In the next gaming post , i will show how to do complex Java games using 3D engines and/or Java3D.

The game scenes are entirely configurable , in this release i have made 11 scenes (from 0 to 10) but you can edit the property file to add million of scenes without the need to change any line of code in the game.

Download URL:

http://sourceforge.net/projects/balloongame/

Saturday, September 17, 2011

Performance Tuning of ATG application

Before you start troubleshooting ATG application issue, you may do the following:
-Get the problem definition.
-Gather all possible information about it like affected transaction , when they detect it on load or with single page, in certain time or all over the day, in one managed server or all servers , when it start to occur , what changed before that time, any applied patch or new driver, ...etc...
-Get all possible tools and methods ready to use.
-Get the application code in-hands.
-Start building your investigation plan.
-Plan can have trial elements, gather information elements, possible fix elements and permanent fix/conclusion.
-If the issue is in the production env, try to replicate it in other env, so you can try fixing it without any possible business impact.

Here is a guide for your investigation and setup of the production environment for ATG application build from my experience plus ATG documentations :

1) ATG Application side Recommendations:

-Enabling liveconfig Settings:
When you’re ready to deploy your Nucleus-based application
in a production environment, enable the settings in the liveconfig configuration layer. This layer overrides many of the default configuration settings with values that are more appropriate for a deployed site. For example, the liveconfig configuration layer improves performance by reducing error checking and detection of modified properties files.

To enable liveconfig, you can use the –liveconfig argument for runAssembler
Or
or add the following line to the WEB-INF/ATGINF/
dynamo.env file in the atg_bootstrap.war module of your EAR file:
atg.dynamo.liveconfig=on

a) Disabling Checking for Changed Properties Files:
This property controls whether or how often ATG rereads .properties files or .java files the next time that an instance of a given component is created (non-global)
The default is 1000. This feature is useful during development, but we recommend disabling it once a site goes live for better performance. The value -1 disables the reloading of properties and .java files altogether.

b) Disable Performance Monitor:
/atg/dynamo/service/PerformanceMonitor) can be used to gather statistics about the performance of specific operations in ATG components.
You can disable the Performance Monitor by setting its mode property to 0:
mode=0
The Performance Monitor is disabled in the liveconfig configuration layer.

c) Adjusting the pageCheckSeconds Property:
ATG’s Page Processor compiles JHTML pages into .java files (JSP compilation is handled by your application server). The page processor, located at /atg/dynamo/servlet/pagecompile/PageProcessor, checks for new Java Server Pages that need to be compiled. You can improve performance by increasing the Page Processor’s pageCheckSeconds property. The page compile servlet uses this property value to determine whether to check for new Java Server Pages that need to be recompiled. If a request occurs within this time interval (measured in seconds) for the same page, ATG will not check the date on the file. This improves performance in serving pages.
A value of 0 causes ATG to check for new pages on each request. The default value is 1. The liveconfig value is 60.

-Fine-Tuning JDK Performance with HotSpot
Refer to Oracle Hot Sport performance tuning for more details.

-Configuring for Repositories:
a) Enable Caching:
Specify the correct values of cache according to your data size, example of this calculation:



b) Setting Cache Modes:
Select the proper cache mode
Remember that if you use locked mode caching, you must also enable lock manager components.

c) Populating Caches on Startup:
This benefit may come at the cost of slower startup times.
You can pre-populate caches in a SQL Repository by using tags in a repository definition file.

d) Configuring Repository Database Verification for Quicker Restarts:
By default, each SQL Repository component verifies each of the tables in its database on startup with a simple SQL query. These verification queries can slow the ATG startup routine.
you may wish to set the updateSchemaInfoCache property to true in your atg.adapter.gsa.GSARepository components, such as /atg/dynamo/service/jdbc/ProfileAdapterRepository.

e) Configure proper caching timings:
**item-cache-timeout :
This attribute defines how long (in milliseconds) a repository item can exist in the item cache without having been accessed before it needs to be reloaded from the database. Effectively, there is a "last touched" timestamp associated with each item cache entry; if the time since that item was last touched is greater than the item-cache-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

**item-expire-timeout :
This attribute defines how long (in milliseconds) a repository item can remain in the item cache before it needs to be reloaded from the database. Effectively, there is a "time loaded" timestamp associated with each item cache entry; if the time since that item was cached is greater than the item-expire-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

**query-expire-timeout :
This attribute is the same as the item-expire-timeout but for query cache entries. This attribute defines how long (in milliseconds) a query can remain in the query cache before it needs to be reloaded from the database. Effectively, there is a "time loaded" timestamp associated with each query cache entry; if the time since that entry was cached is greater than the query-expire-timeout setting, then it's properties are loaded from the database instead of the in-memory cache, which is then updated with the values from the database.

Note that the query-expire-timeout attribute only applies when you have query caching enabled.

-Setting Logging Levels :
If you want to disable logging entirely, or specify different logging levels, you can do that in the
GLOBAL.properties file. For example:
loggingError=true
loggingWarning=true
loggingInfo=true
loggingDebug=false

**Your application code must follow the standard in checking on the level before logging the log messages.

-Limiting Initial Services for Quicker Restarts
This is configured using initialServices property of the /atg/dynamo/Initial component.

-Disabling Document and Component Indexing
The ACC creates and maintains indexes of documents and components. For sites with large numbers of documents or components, indexing can take time and CPU resources. Once your site is deployed and relatively stable, you may want to limit or eliminate the indexing of documents or components.
The document and component indexes are maintained incrementally once built, and are rebuilt completely once a day at 1 a.m. by default. An index is rebuilt at startup only if it does not exist at all.
You can selectively exclude portions of the document tree from indexing by adding absolute pathname prefixes to the excludeDirectories property of the /atg/devtools/DocumentIndex component.
The same is true for component indexing, but the component is /atg/devtools/ComponentIndex instead. To improve performance on a live site, you can turn off all document and component indexing by setting the enabled property of the DocumentIndex and ComponentIndex components to false.

-Compress Content:
Compress pages (remove white-spaces) and static content (pictures, JS, CSS) which will speed up download time from the browser in your Web Server.
Compressing HTML/JavaScript/CSS/XML/JSON content can significantly reduce response times. GZIP reduces the size of responses by between 50% and 80%, depending on the type of content. Only turn on GZIP compression for text/html, text/plain, text/json, text/css, and application/x-javascript mime types.
To verify that gzip is being used, “Accept-Encoding: gzip,deflate” should show up in the request header and “Content-Encoding: gzip” should show up in the response header.

-Re-structure your page to have:
*Move .css to the top of all pages.
*Move script files to the button of all pages whenever possible.
*Move all embedded script into external files.

-Ajax Cache:
Cache your ajax response whenever possible to speed us the user experience whenever the results can be cached.

-Pre-Compiling JSPs:
Might slow deployment/server start-up but will speed up the 1st page request.

-Session Stickiness:
It’s important that session stickiness is working properly. Not having it working could result in sessions being continually restored after each request.

-Keep ATG Patched:
When possible, the latest version of ATG should be used. As cumulative patches are released, they should be applied.

-HTTP Connection Reuse:
Using the “Keep-Alive” header allows browsers to reuse the same TCP connection for multiple request/response pairs. Not re-establishing TCP connections for each HTTP request/response helps Reduce network traffic, Reduce the load on the SSL accelerator and Improve performance due to not having to setup and tear down TCP connections for each HTTP request/response.
A good Keep-Alive value for CSC is 300, which is 5 minutes.
Both Internet Explorer 6 and 7 forcibly terminate TCP connections after one minute, regardless of what Keep-Alive is set to. See http://support.microsoft.com/kb/813827 for a workaround.

2) Server\JVM\Operating system Configurations:
-Setup JDBC connections with number matching the expected concurrent users of the site.
-Setting the JTA timeout to 120 seconds whenever possible.
-Increase the max concurrent open files by the operating system.
-Compress the web application output.
-Configure Max Thread Stuck to a propert time to catch where the threads are usually stucked..

3) Using Performance measure tools:

A) ATG built-in Features:

1) Performance monitor:
*Adding PerformanceMonitor Methods to your Code
To enable the Performance Monitor to monitor a section of your Java code:
1. Import the atg.service.perfmonitor.* package.
2. Declare an opName parameter to label the section of the code. This parameter is displayed in the Performance Monitor page under the Operation heading.
3. (Optional) Declare a parameter name if you want to gather data on individual executions of an operation.
4. Call the startOperation method at the beginning of the operation whose performance you want to be able to measure.
5. Call the endOperation method at the end of the operation whose performance you want to be able to measure.
6. Optionally, call the cancelOperation method if an exception occurs. This causes the results of the current execution to be ignored.

These methods can be nested with different or the same opNames.
Example:
PerformanceMonitor.startOperation(opName, parameter);
try {
... code to actually render foo.jsp
} catch (Exception e) {
PerformanceMonitor.cancelOperation(opName, parameter);
exception = true;
} finally {
if (! exception)
PerformanceMonitor.endOperation(opName, parameter);
}

*Performance Monitor Modes:
You can set the Performance Monitor’s operating mode by setting the mode property of the component at /atg/dynamo/service/PerformanceMonitor:
disabled 0 (default)
normal 1
time 2
memory 3
You should use 2=Time to get accumulated results, also try to enable it after warm up the site to exclude extreme reading for loading caches ,...etc.

*View the Results:
You can view the information collected by the Performance Monitor on the Performance Monitor’s page
of the Dynamo Administration UI at:
http://hostname:port/dyn/admin/atg/dynamo/admin/en/performance-monitor.jhtml

2) Using the VMSystem Component :
The ATG component located at /VMSystem provides a way for you to access the Java memory manager.
You can monitor the status of the Virtual Machine and call methods on it. An interface to the VMSystem component is included in the Dynamo Administration UI at:
http://hostname:port/dyn/admin/nucleus/VMSystem/
From this page, you can conduct the following VM Operations:
• Perform garbage collection
• Run finalizations
• Show memory information
• List system properties
• List thread groups
• List threads
• Stop the VM

3) Sampler:
When testing your site, it is useful to automatically sample performance to understand throughput as a function of load. ATG includes a Sampler component at /atg/dynamo/service/Sampler.
Starting the Sampler You can start the Sampler component by opening it in the ACC and clicking the Start button.
You can also start the Sampler component from the Dynamo Administration UI by requesting this URL:
http://hostname:port/dyn/admin/nucleus/atg/dynamo/service/Sampler
The first time you request this page, ATG instantiates the Sampler component, which begins recording statistics.
You can configure ATG to start the Sampler whenever ATG starts by adding the Sampler to the initialServices property of the /atg/dynamo/service/Initial component:

The Sampler outputs information to the file /home/logs/samples.log. For each system variable that it samples, it records the following information in the log file:
• the current value
• the difference between the current value and the value recorded the last minute
• the rate of change of the value
You can adjust values recorded by the Sampler, but the default set is comprehensive in monitoring ATG request handling performance.

B) Log Files:

* Access Logs:
You may enabled access logs on your application sercer or web server to ensure that the time is really consumed inside your application not in network traffic (download time)..

* Application Logs:
Application logs could point to system calls or external systems timeout or DB issues , exceptions , .... alot of useful information could be detected from the application logs.
You might see a “server not responding” message or an OutOfMemory error

C) Thread Dump:
Thread dumps can be useful to see where these threads are
waiting. If there are too many threads waiting, your site’s performance may be impaired by thread context switching. You might see throughput decrease as load increases if your server were spending too much time context-switching between requests. Check the percentage of System CPU time consumed by your JVM. If this is more than 10% to 20%, this is potentially a problem. Thread context switching also depends
in part on how your JVM schedules threads with different priorities

Thread dumps can be taken from Admin console of the command line.
Thread dumps could also point easily to deadlocks and infinite loops and other issues related to bad buggy code.

D) Garbage Collection:
Check the JVM parameters that affect Garbage Collection including gc policy, try to set max and min heap size the same value.
Check the garbage collections Logs; typically when you see spikes or abnormal behaviors, the most important is the full garbage collection runs.

*Phases that stop the threads with the parallel garbage collection algorithm:
tail -f | grep 'Full'
*Phases that stop the threads with the Concurrent Mark Sweep algorithm:
tail -f | grep -E '(CMS-initial-mark)|(Rescan )|(concurrent mode
failure)|(Trying a full collection)|(promotion failed)|(full)|(Full)'

If excessive pausing is noticed, one of several things could be wrong:
- JVM arguments may need to be tuned
- There may be a memory leak
- Repository/droplet/atg.service.cache.Cache caches may be over-utilized
- Load balancing could be not working properly, which would result in more sessions than normal hitting one instance.

While thread pauses are a normal part of garbage collection, excessive pauses must be minimized.

E) Cron Jobs:
Check the running cron-jobs on the server and try to only setup them to run during non-load hours of the servers.

F) Profiling Tools:
-Netbeans Profiler
-JProfiler
-Eclipse TPTP
Any other profiling tools that give you detailed information about consumed in any operation and memory tracing ,...etc.

G) Load testing Tools:
*) URLHammer (ATG load tool):
To run the URLHammer program:
1. Set your CLASSPATH to include the directory /DAS/lib/classes.jar.
2. Run the following command:
java atg.core.net.URLHammer [arguments]
For example:
java atg.core.net.URLHammer http://examplehost:8840/ 5 10 -cookies
This creates five different threads, each of which represents a separate session that requests the specified
URL 10 times (50 requests total).

You can also run a script by editing the format yourself or by using RecordingServlet to create it..

*) Apache JMeter (open source)
*) HP Load Runner (commercial)

H) Client Side Performance Tools:
-Firebug (FF plugin)
-Fiddler (Standalone or plugin)
-DynaTrace (Standalone or plugin)
The most important is to identify if a certain resources (esp outside your domain) is taking much of the time.
Invalid configuration also could be detected , pointing to another env.
JS performance might be a reason for end-user bad performance.

I) Operating system performance Tools:
Monitoring System Utilization Use a program like top (on Solaris), the Windows Performance Monitor, or a more sophisticated tool to keep track of information like:
• CPU utilization
• paging activity
• disk I/O utilization
• network I/O utilization
*If you are getiing Can't create native thread exception , you should know that your main system memory is low , as each native thread reserve around 1 MB of memory for its stack trace.
*You can detect a file descriptor leak in two different ways:
• You may notice a lot of IOExceptions with the message “Too many open files.”
• During load testing, you periodically run a profiling script, such as lsof (on UNIX), and you notice that the list of file descriptors grows continually.

J) DB performance tuning:
- Take DB snapshots and analysis them.
- Take the most consuming sql and do execution plan for them, possibly you could find a missing index.
- You may need a DBA to analyze possible DB issue.
- Enable JDBC logging and retrieve the SQL queries and try to optimize them outside the application (debug level need to be set to 15, you can also retrieve this queries from DB monitor tools)

K) Adjusting the FileCache Size :
ATG’s servlet pipeline includes servlets that are used for JHTML pages, and which use a FileCache component to store files that ATG has read from disk, so that subsequent accesses for those files can be delivered directly from memory instead of being read from disk. Using the FileCache component improves performance by reducing disk accesses. For maximum performance, you want the FileCache to be large enough to hold all the files that ATG serves frequently.
Set the totalSize property of this component at:
/atg/dynamo/servlet/pipeline/FileCache
to an appropriate value, measured in bytes, such as the following:
# size in bytes (2 million bytes)
totalSize=2000000

One approach in sizing the FileCache is to batch compile the entire document root and set the file cache to the resulting size. Make sure, however, that you account for the size of your FileCache when
you set the size of your JVM. You can preload the FileCache by creating a script that accesses every page on your site and running the script on startup.
You can view statistics on how the file cache is used, as well as the contents of the FileCache in the Dynamo Administration page at
hostname:port/dyn/admin/nucleus/atg/dynamo/servlet/pipeline/FileCache.

L) Code optimization:
You may scan the code in the high transaction scenarios, with special care to the pipelines and component scopes to identify possible performance issues..
You may use tools for code optimizations.
Some finding may need changes to component scope (the global is best performing being loaded only once, while the request scoped is initialized per each request, so you need to minimize it)
Avoid resolving the components from the code, instead using ATG property files to inject them , in case you do not have that ability like derived properties , you may have static method to get a reference to them (if global components).
Follow the code standards and best practices..
Once possible reason for application bad performance is not following the logging best practices by checking the log level before logging the message.

M) Use Cache Droplet in your JSP pages:
Cache Droplet caches content that changes infrequently used especially if it includes a lot of processing or DB interactions (Component /atg/dynamo/droplet/Cache)
**Required Input Parameters key
Lets you have more than one view of content based on a value that uniquely defines the view of the content. For example, if content is displayed one way for members and another for non-members, you can pass in the value of the member trait as the key parameter.
**Optional Input Parameters
-hasNoURLs
Determines how cached URLs are rendered for future requests. By setting hasNoURLs to false, you specify that subsequent requests for the cached content causes URLs to be rewritten on the fly, assuming URL Rewriting is enabled. A setting of false for hasNoURLs causes URLs to be saved and rendered exactly as they are currently (without session or request IDs) regardless of whether URL rewriting is enabled.

-cacheCheckSeconds
The interval after content is cached until the cached is regenerated. If omitted, the interval is set from the defaultCacheCheckSeconds property in the Cache servlet bean’s properties file.

**Open Parameters output
The code enclosed by the output open parameter is cached.

**Clearing the cache:
You can determine how often data is flushed for a given Cache instance on a JSP or for all instances of Cache. To remove cached content associated with a particular instance of Cache, set the cacheCheckSession input parameter in the Cache instance to the frequency by which associated data should be expired. If you omit this parameter, the Cache.defaultCacheCheckSeconds property is used (default value is 60 seconds) .
The Cache.purgeCacheSeconds property determines how often content cached by any Cache servlet bean is flushed. The default is 21600 seconds (6 hours). Cache purging also occurs when a JSP is removed or recompiled.

N) Hardware limited capacity:
This case for development environment where you may consider moving one application outside this box, like the DB , Merch , ...etc..
Also you can shutdown the Merch if you do not need it up and running all the time.
Another trick -strictly for Dev box- is decreasing session timeout into 5 minutes or decrease the thread reserved memory usage.


O) ATG recommended Check List:
The following checklist can help you identify the most common sources of performance problems:
• Have you properly configured memory for your Java Virtual Machines? Have you set your -Xms and -Xmx arguments the same? Do all ATG heap sizes fall within the limits of physical memory?
• Has one or more servers stopped responding? There could be a number of causes, including a Java deadlock.
• Are you seeing many IOExceptions with the message “Too many open files”? You may have a file descriptor leak.
• At maximum throughput, look at the CPU utilization, database CPU utilization, I/O activity, and paging activity.
• If CPU utilization is low, then you may have an I/O or database bottleneck.
• If CPU utilization is high, then the bottleneck is most likely in the application code. Use a performance profiling tool to try to locate bottlenecks in the code. Review your code to make sure it uses good Java programming practices.
• If paging is occurring, adjust the memory allocated to your Java Virtual Machines.
• Look at the I/O and CPU utilization of the database. If utilization is high, database activity is probably slowing down the application.
• Are you receiving page compilation errors? You may not have enough swap space for page compilation.


Reference: ATG Platform documentation set : Version 9.1 - 7/31/09 and others..
For More information, refer to Java EE 7 performance tuning and optimization book: The book is published by Packt Publishing: http://www.packtpub.com/java-ee-7-performance-tuning-and-optimization/book

Wednesday, September 14, 2011

ATG Made Easy - part 6

ATG Commerce
2 versions: ATG Consumer Commerce is used for developing standard business-to-consumer (B2C) online stores. ATG Business Commerce is used for sites oriented more toward business-to-business (B2B) uses.

1) Product Catalog & Custom Catalog:
The product catalog is a collection of repository items (categories, products, media, etc.) that provides the organizational framework for your commerce site. ATG Commerce includes a standard catalog implementation, based on the ATG SQL Repository, that you can use or extend as necessary.

The structure is built on Catalog --> Category --> Product --> Sku
Where you can modify the features of each level.
The most important is to have the parent flag in the top categories
Each level has parent/child attributes, eg. parentCategories, childSku's for product level.
You can have a linked template to display the element using it.
Some dropletes are dedicated on loading these elements: ItemLookupDroplet (generic) versus CategoryLookupDroplet, ProductLookupDroplet and SKULookupDroplet.
They take id as input and product the item/elements as output.

MediaLookupDroplet for Media , 2 types exist: Media Internal and External (internally referenced versus URL/File)
The data property is either binary (for media-internal-binary items) or text (for media-internal-text items)

ATG catalogs use several batch and dynamic services to perform catalog updates and to verify catalog relationships. These services are collectively referred to as the Catalog Maintenance System (CMS). The CMS updates property values that enable navigation and hierarchical search. It also verifies the relationships of catalog repository items and properties.
**Batch Services : as
• AncestorGeneratorService
• CatalogVerificationService
• CatalogUpdateService
• CatalogMaintenanceService
Each service creates a global lock at the start of execution, using the global lock management, and releases it upon completion. This prevents services that use the same lock name from executing simultaneously on the same server, or other servers in the cluster.
**Dynamic Services : as
• CatalogChangesListener
• CatalogCompletionService
• StandardCatalogCompletionService

Note that some of these services are available only for custom catalogs; others for both standard and
custom catalogs.

2) Order Management:
A session scoped component which is ShoppingCart (atg/commerce/ShoppingCart) manage the current order (+last order and saved orders), order stored in memory and persisted frequently..
You can configure and customize it using orderrepositiory.xml (where you can define its components, cache size and lock mode).
Once Sku is added to the order-->it become CommerceItem that is related to ShoppingGroup that is related to ShippingAddress and ShippingMethod.
Once order is submitted , it become in incomplete status ---> further processing --> fulfilled.
In case of B2B, order is hold for the needed approvals.
Once good feature exist in ATG Commerce, is the ability to have scheduled order.

Payment methods include: Credit card, Store credit, gift certificate, invoice request (in B2B)

CartModifierFormHandler: Request scope
A form handler to handle adding items to order, remove items, change quantity, move the order forward (payment, shipping info),

StoreCartFormHandler:
Handles continue shopping, expressCheckout, update, ....etc...

*Business Layer:
-CommerceItemManager:
Create, add to order, remove from order..
Once sku is added to the order : sku id--> catalog ref Id in CommerceItem object, quantity , priceInfo , paymentGroup and shippingGroup are taken as well.

-OrderManager:
load/save, create , allocate order to payment group.
These operations are handled by pipelines as mentioned before when we discuss the pipelines.
-OrderTools:
Provides low level raw operations..
-PurchaseProcessHelper:
Centralized functionality for purchase operations.

*Best Practise for Order Updating:
1.Acquire lock-manager write lock on profile id from the /atg/commerce/order/LocalLockManager
2.Begin Transaction
3.Synchronize on the Order object.
4.Modify Order
5.Call ((OrderImpl) pOrder).updateVersion();
6.Call OrderManager.updateOrder()
7.Release Order synchronization
8.End Transaction
9.Release lock-manager write lock on profile id from the /atg/commerce/order/LocalLockManager


3) Purchasing and Fulfillment Services :
ATG Commerce provides tools to handle pre-checkout order-processing tasks such as adding items to a shopping cart, ensuring items are shipped by the customer’s preferred method, and validating credit card information. The system is designed for flexibility and easy customization; you can create sites that support multiple shopping carts for a single user, multiple payment methods and shipping addresses.
You can integrate with third-party authorization and settlement tools such as Payflow Pro, CyberSource, and TAXWARE.

4) Inventory Management :
The inventory framework facilitates inventory querying and inventory management for your site.
InventoryManager
The InventoryManager is a public interface that contains all of the Inventory system functionality. Each method described below returns an integer status code. All successful return codes should be greater than or equal to zero, all failure codes should be less than zero. By default, the codes are:
INVENTORY_STATUS_SUCCEED=0 There was no problem performing the operation.
INVENTORY_STATUS_FAIL=-1 There was an unknown/generic problem performing the operation.
INVENTORY_STATUS_INSUFFICIENT_SUPPLY = -2 The operation couldn’t be completed because there were not enough of the item in the inventory system.
INVENTORY_STATUS_ITEM_NOT_FOUND = -3 The operation could not be completed
because a specified item could not be found in the inventory system.

ATG Commerce includes the following implementations of the InventoryManager out of the box.
• AbstractInventoryManagerImpl
• NoInventoryManager
• RepositoryInventoryManager
• CachingInventoryManager
• LocalizingInventoryManager

Preventing Inventory Deadlocks
InventoryManager includes the acquireInventoryLocks and releaseInventoryLocks methods.
acquireInventoryLocks acquires locks for the inventory items that apply to the given IDs.
releaseInventoryLocks releases locks for the inventory items that apply to the given IDs.

5) Pricing Services :
ATG Commerce pricing services revolve around pricing engines and pricing calculators.
**The pricing engine determines the correct pricing model for an order, individual item, shipping charge, or tax, based on a customer’s profile.
**The pricing calculator performs the actual price calculation based on information
from the pricing engine.

Pricing engines are responsible for three tasks:
• Retrieving any promotions that are available to the site visitor.
• Determining which calculators generate the price.
• Invoking the calculators in the correct order.

Pricing calculators are responsible for the following:
• Looking up the price in the catalog by priceList.
• Invoking a qualifier service that identifies the objects to discount.
• Using information they receive from the engines and from the qualifier service to
perform the actual process of determining prices.

By default, ATG Commerce can perform dynamic pricing for the following types of pricing object:
• Items. Each item has a list price that can be specified in the listPrice property of the Product Catalog repository.(Note that an “item” is a CommerceItem, which represents a quantity of a SKU or a product).
• Orders.
• Shipping price.
• Tax.

Terminology:
-Qualifier : This is a service that interprets a PMDL rule and decides what, if
anything, may be discounted. The term qualifier also refers to the first part of a PMDL rule. This rule defines when something can receive a discount.
- Target : The second part of a PMDL rule is called the target. This rule
defines which part of an order receives a discount.

4 Types of PriceInfo objects:
OrderPriceInfo,ItemPriceInfo, ShippingPriceInfo, and TaxPriceInfo

How this works?
1-Invokation to price engine from business-layer logic, such as a PriceItem servlet
bean in a page or from the ATG Commerce PricingTools class.
2-Pricing engine applies its configured precalculators. A precalculator modifies a
price without using any associated promotions.
3-Pricing engine accesses the current customer’s profile and retrieves any
promotions listed in the activePromotions property of the profile
4-Pricing engine builds a list of global promotions and concatenate them.
5-Pricing engine applies promotions by priority (each promotion has pricingCalculatorService property that specifies the calculator that the system must use to apply it)
6-The pricing engine applies its configured PostCalculators.
7-The pricing engine modifies the PriceInfo object of the object being discounted.

-When building a product catalog, you must decide whether your site requires dynamic product pricing and, if so, how granular you need it to be. Using dynamic pricing on a product page can cause a significant decrease in performance compared to using static pricing..

-With static pricing, each item in the catalog has a list price stored in the listPrice property of the catalog repository.
Volume Pricing: Bulk 100 at 10, 200 at 9, ...etc..
or Tiered: 1st 100->10 , 2nd 100 -->9 , ...
Static uses 2 droplets: PriceDroplet and PriceRangeDroplet.

**PriceEngine:
atg.commerce.pricing.PricingEngine is the main interface for interacting with the
atg.commerce.pricing package. Extensions of this interface describe objects that calculate a price for a specific class of object. For example, OrderPricingEngine extends PricingEngine and calculates prices for orders passed to it.

All PricingEngine implementations process promotions. The PricingEngine interface itself contains only one method, getPricingModels, which extracts a collection of promotions from an input profile. (Price Model can be customized by priceModel.xml).
The PricingEngine interface itself does not describe any functionality other than the promotion extraction API because of the wide range of information that different PricingEngine implementations might require to calculate a price for their specific class of object. For example, the ItemPricingEngine implementation needs one set of input parameters, while the OrderPricingEngine needs a different set.

The individual extensions of the PricingEngine interface contain the API methods for generating a given type of price. There is a Java object type for each type of price that is generated. For example,atg.commerce.pricing.OrderPricingEngine inherits the promotion extraction API from PricingEngine and defines one new method, priceOrder, to generate a price for an order in a given context.

ATG Commerce provides the following four extensions of the main PricingEngine interface:
• atg.commerce.pricing.ItemPricingEngine
Provides a price for atg.commerce.order.CommerceItem objects.
• atg.commerce.pricing.OrderPricingEngine
Provides a price for atg.commerce.order.Order objects.
• atg.commerce.pricing.ShippingPricingEngine
Provides a price for atg.commerce.order.ShippingGroup objects.
• atg.commerce.pricing.TaxPricingEngine
Determines tax for atg.commerce.order.Order objects.

PricingTools Class
The atg.commerce.pricing.PricingTools class performs a variety of pricing functions for different types of pricing engines. It also has a number of static, currency-related methods for use by all pricing engines.
The PricingTools class is the main way that business-layer logic interacts with the pricing engines and the other classes in the atg.commerce.pricing package.

The properties of PricingTools are as follows:
• itemPricingEngine: The pricing engine that calculates prices for items, both
individually and in groups. An item is identified as a set quantity of a SKU or product.
• orderPricingEngine: The pricing engine that calculates prices for orders. Typically,
the price is just the sum of the prices of the items in the order. However, the order
might be discounted separately from the constituent items.
• shippingPricingEngine: The pricing engine that calculates prices for shipping
groups. An order contains one or more shipping groups when the contents of the
order require shipping for delivery. An order has no shipping groups when it is
delivered online, and shipping is therefore not calculated for this type of order.
• taxPricingEngine: The pricing engine that calculates tax for orders. Tax is calculated
on the order total.
• roundingDecimalPlaces: Specifies the number of decimal places to which the an
input price is rounded . This property is used by the round, roundDown and
needsRounding methods.

Important Methods:
priceEachItem, priceItem, priceItemsForOrderTotal, priceOrderForOrderTotal, priceOrderTotal, priceShippingForOrderTotal, priceTaxForOrderTotal, needsRounding, round.

The Pricing Servlet Beans : You can insert on site pages as required and use to
perform dynamic pricing.
• AvailableShippingMethods Servlet Bean
• ItemPricingDroplet Servlet Bean
• PriceEachItem Servlet Bean
• PriceItem Servlet Bean
• PriceDroplet Servlet Bean
• ComplexPriceDroplet Servlet Bean

*Price Lists:
Price Lists allow you to target a specific set of prices to a specific group of customers.
The PriceListManager class maintains the price lists. A price may be retrieved from the PriceListManager from a given price list by product, by SKU, or by a product/SKU pair.
The most important method in PriceListManager is getPrice. This is used during pricing of an order to get the correct price for a given product/SKU pair.

It is configured using priceList.xml

6) Targeted Promotions :
Business managers can use ATG Commerce promotions to highlight products and offer discounts as a way of encouraging customers to make purchases.

Promotions 3 types: ItemDiscount, OrderDiscount and ShippingDiscount.

Pricing Model Description Language (PMDL): rule describing the conditions under which this promotion should take effect. The rule is created in the ATG Control Center using the interface designed for promotion creation.

7) Order Management Web Services :
All order management web services are included in commerceWebServices.ear in the
orderManagement.war web application.

Reference: ATG Platform documentation set : Version 9.1 - 7/31/09

Tuesday, September 13, 2011

ATG Made Easy - part 5

ATG Personalization Features:

One of the power features of ATG is the personalization module, here is a quick list of the available features:

1) Internal and External User Profiles:

The default external user profile repository is /atg/userprofiling/ProfileAdapterRepository,
which is defined by the userProfile.xml file located in <ATG9dir>\DPS\config\profile.jar. Each
ATG application that adds properties to the external user profile stores its userProfile.xml file in an
ExternalUsers sub-module.
Internal profiles are stored in the /atg/userprofiling/InternalProfileRepository, defined by the
internalUserProfile.xml file in <ATG9dir>\DPS\InternalUsers\config\config.jar.
A parallel set of database tables also exists for internal user profiles. Where the user item in the
ProfileAdapterRepository references the dps_user table, the user item in the
InternalProfileRepository points to a dpi_user table, and so on.

You can extend/replace and customize the userProfile.xml using the same way as any repository.

In ATG, you can configure the profile to use SQL or LDAP or both.
By changing : /atg/userprofiling/ProfileTools into :
profileRepository=/atg/adapter/ldap/LDAPRepository

The LDAP is based on view, object and classes:
<!-- user view -->
<view name="user" default="true">
.....
<!-- object classes -->
<object-class>top</object-class>
<object-class>person</object-class>
<object-class>organizationalPerson</object-class>
<object-class>inetorgPerson</object-class>
<!-- properties -->
<property name="login" ldap-name="uid" data-type="string" required="true">
<attribute name="unique" value="true"/>
</property>
<property name="password" ldap-name="userpassword" data-type="string"
required="true"
editor-class="atg.beans.PasswordPropertyEditor"/>

Whenever a user accesses a site that uses the Personalization module, two different mechanisms are used
to track the user’s actions:
• A session is created for the user, and is maintained either through a cookie or through
URL rewriting.
• The user is associated with a profile.
To change the secret key that the Personalization module uses to hash the user ID cookie, edit the
following property of /atg/userprofiling/CookieManager:
cookieHashKey=

Profile Form Handlers:

-handleCreate: Creates a new permanent profile and sets the profile attributes to the values entered in the form.
-handleUpdate Modifies the attributes of the current profile.
-handleLogin Uses the login and password values entered by the user to associate the correct profile with that user.
-handleChangePassword Changes the password attribute of the profile to the new value entered by the user.
-handleLogout Resets the profile to a new anonymous profile and optionally expires the current session.


Access Control
Control access to some or all of the pages on the site.
AccessControlServlet (/atg/userprofiling/AccessControlServlet):

Configurations:
$class=atg.userprofiling.AccessControlServlet
enabled=true
# Nucleus path of the Profile object
profilePath^=ProfileRequestServlet.profilePath
# List of mappings between paths and AccessController objects. If a
# path refers to a directory, all the documents in that directory and
# its subdirectories will be protected by the given AccessController.
accessControllers=\
/docs/members=/your/path/MemberAccessController,\
/docs/members/preferred=/your/path/PreferredMemberAccessController
# List of "access allowed" event listeners
# accessAllowedListeners=
# List of "access denied" event listeners
# accessDeniedListeners=
# The URL to redirect to if access is denied. If the AccessController
# supplies its own deniedAccessURL, it will overwrite this value.
deniedAccessURL=http://yourserver/noaccess.html


*RuleAccessController
This implementation of AccessController performs access control based on a set of rules, specified via the service’s ruleSetService property. For example, suppose there is a RuleSetService named
FemaleRuleSetService, configured with the following rule set:
<ruleset>
<accepts>
<rule op=eq>
<valueof target="Gender">
<valueof constant="female">
</rule>
</accepts>
</ruleset>
Set the ruleSetService property of the Access Controller to point to
FemaleMembersRuleSetService. The user will be allowed access only if she is in the Female profile
group. Here is the example configuration:
$class=atg.userprofiling.RuleAccessController
enabled=true
# Rules used to determine whether access should be allowed
ruleSetService=/your/path/rules/FemaleRuleSetService
# URL to redirect to if access is denied
deniedAccessURL=http://yourserver/femaleAccessOnly.html


2) Targeting Content:

Creating Rules for Targeting Content:
Example:
<rule op=eq name="Rubber sector">
<valueof target="industrySector">
<valueof constant="rubber">
</rule>
Type:
Accept Rules
Reject Rules
Sorting directives
<ruleset>
<accepts>
<rule ...> ... </rule>
...
</accepts>
<rejects>
<rule ...> ... </rule>
...
</rejects>
<includes>
<ruleset src=...> ... </ruleset>
...
</includes>
<sortby>
<sortbyvalue ...>
...
</sortby>
</ruleset>


Setting Up Targeting Services:
To set up a RuleSetService for your rule set, create an atg.targeting.RuleSetService component.
This component can reference a rules file, or it can itself include your targeting rules as a property

So either
-rulesFilePath:If your Rule Set Service refers to a rules file, set this property to the file path of the rules file. This path can
be an absolute path or a relative path starting from your <ATG9dir>/home directory.
-ruleSet=xml rules structure

** Targeter Example: calling slot from targeter

<dsp:droplet name="/atg/targeting/TargetingFirst">
<dsp:param name="targeter" bean="/atg/registry/slots/aricleSlot"/>
<dsp:param name="howMany" value="1"/>
<dsp:oparam name="output">
<dsp:a href="articleDetails.jsp">
<dsp:param name="itemId" param="element.id"/>
.....
.....other retrived parameters ....
.....
</dsp:a>
</dsp:oparam>
</dsp:droplet>

Targeted E-mail:
You can use the Targeted E-mail services included with the Personalization module to compose and deliver e-mail using the same profile groups and targeting rules you use to deliver content on your Web site.

You create targeted e-mail using the:
atg.userprofiling.email.TemplateEmailInfoImpl class.
This class draws the message body of an e-mail from a page template, invokes a MessageContentProcessor component to process the content, and then passes the resulting JavaMail object to the TemplateEmailSender component, which sends the message. The properties of a TemplateEmailInfoImpl object store values that are specific to an individual e-mail campaign, so you should create a separate instance of this class for each campaign.
*key properties of the TemplateEmailInfoImpl class:
templateURL
messageFrom
messageTo
contentProcessor:MessageContentProcessor responsible for processing the message content; Default: /atg/userprofiling/email/HtmlContentProcessor another valid value is SimpleContentProcessor
...

The HtmlContentProcessor can be further configured according to the needed formatting.


3) Scenario Module:
Scenarios are event-based i.e. what does user/system do? but targeters are knowledge-based i.e. what does the user info/snapshot contain.

The configuration file scenarioManager.xml is the place where information common to all scenario servers is specified. This file uses the Process Manager DTD, located in
<ATG9dir>\DSS\lib\classes.jar. The scenarioManager.xml file
A cluster of ATG servers must always contain the following:
• exactly one process editor server
• zero or more global scenario servers
• zero or more individual scenario servers

The main Nucleus component responsible for scenario operations is located at /atg/scenario/ScenarioManager. To examine the scenarios handled by this service, point your Web browser to the ATG Dynamo Server Admin page at:
http://localhost:port/dyn/admin/nucleus/atg/scenario/ScenarioManager/

You can defining Access Control for a Scenario as well..
You can bound the scenario to events like :
Collective events:
• InboundEmail Event
• Shutdown Event
• Startup Event
• GSAInvalidation Event
Individual events:
• Login Event
• Logout Event
• Register Event
• AdminRegister Event
• StartSession Event
• EndSession Event
......etc.

The scenario is componsed of actions , Scenario actions are implementations of the atg.process.action.Action interface or direct extends ActionImpl class.
The main methods are:
-initialize (map of parameters)
-configure (Config object) , you can cast to your customized configuration object according to the action definition xml; the config object extends Generic Service and contains getters and setters for all configuration elements.
-executeAction(ProcessExecutionContext context)
Where you can have access to request, response, user, events, parameters, properties set by admin for all users ...etc...

Configuration file : ScenarionManager.xml

<action-name>
</action-name>
Yes The logical name of the action as passed to an
action handler.
<action-class>
</action-class>
Yes A Java class that is an implementation of the atg.process.action.Action interface.
<action-configuration>
</action-configuration>
No The Nucleus path of the action’s configuration file.
Most of other elements are optional only action name and class are required.

Default scenario actions:
• Modify Action
• Set Random Action
• Redirect Action
• FillSlot Action
• EmptySlot Action
• Disable Scenario Action
• Record Event Action
• Record Audit Trail Action
• Filter Slot Contents Action
• Add Marker To Profile Action
• Remove All Markers From Profile Action
• Remove Markers From Profile Action
• Add Stage Reached Action
• Remove Stage Reached Action
• E-mail-Related Actions: EmailNotify and SendEmail



4) Using Slots :

Slots are containers that you can use to display and manage dynamic items on your Web site. You use targeting servlet beans to include slots in site pages, and you use scenarios to fill them with content.
Slots are components of class atg.scenario.targeting.RepositoryItemSlot or
atg.scenario.targeting.Slot. A slot component must have a Nucleus address in the folder
/atg/registry/Slots/.
You can create slot components in two ways:
• By manually creating a .properties file
• Through the slot wizard in the ACC

The following is an example of a .properties file for a slot component of class
atg.scenario.targeting.RepositoryItemSlot:
$class=atg.scenario.targeting.RepositoryItemSlot
$description=displays fund news to brokers
$scope=session
generation=0
itemDescriptorName=news
maxRenderSize=3
ordering=1
persistent=true
repositoryName=News
retrieval=1

Most important are:
repository name = source of content, itemDescriptorName also.
The Event Generation option corresponds to generation (property of type int) in the slot component .properties file. For Never, specify 1. For When Empty, specify 0.
Item retrival: 0=static ; 1=rotating ; 2=destructive (displayed only once then removed)
ordering : 0= shuffle, 1=random

A lot of other features still exist in this module.

Reference: ATG Platform documentation set : Version 9.1 - 7/31/09

ATG Made Easy - part 4

11) ATG Search:
The form handler class atg.repository.servlet.SearchFormHandler.
This handler uses the search criteria to return items from one or more repositories, and one or more item types from a given repository. SearchFormHandler is a subclass of atg.repository.servlet.GenericFormHandle,so it inherits properties and handler methods common to all form handlers.

*Supports the following search methods:
• Keyword search
• Text search
• Hierarchical search
• Advanced search
• Combination search

Important properties:
-allowEmptySearch
==>Search proceeds or throw an error when no search criterion is specified. The default value is true.
-allowRefine
==>Include previously entered criteria. default value is false.
-enableCountQuery
==>The items located by a search. default value is false.
-errorURL
==>URL that opens when a search fails. Set this property to the URL of a page that describes the error and provides tools for starting another search.
-successURL
==> URL that opens when a search operation succeeds. This property should point to the URL for the first search results page.
-itemTypes
==> must be defined for the repository specified in the repositories property.
-previouslySubmitted
==>Tracks whether a query that uses this SearchFormHandler occurred during the current session. When a search begins, this property is set to true.
-repositories
==> A comma-delimited list of repositories to include in the search. Specify the full path of each repository to the component.

A) KeyWord Search:
• doKeywordSearch is set to true to enable keyword searching
• keywordSearchPropertyNames specifies one or more single or multi-valued
properties to include in searches. If this property is empty, the form handler searches
all string properties of each repository item, except enumerated or non-queryable
properties.

Logical Operators:
• NOT / !
• AND / &
• OR / |

B) Text Search:
• doTextSearch is set to true to enable text searching.
• textSearchPropertyNames specifies which properties to query.

-allowWildcards (default true , to use *)
-searchStringFormat :
Format that the text should follow.
Each repository component uses this property to specify the text format available to the
database. Available options include:
- ORACLE_CONTEXT: Oracle ConText
- MSSQL_TSQ: Microsoft SQL Server
- SYBASE_SDS: Sybase SpecialtyDataStore
- DBS_TEXT_EXT: IBM DB2 with the Text Extender package

C) Hierarchical Search:
A hierarchical search returns all parent and descendant items of the specified item. A hierarchical search looks in a family of items, starting from a given item level and extending to that item’s descendants.
Each item type must have a multi-valued property whose value is a complete list of its ancestor IDs.
Hierarchical searching restricts the search results to items whose ancestor items include the item specified in the ancestorId property.

• doHierarchicalSearch Set to true.
• ancestorPropertyName Name that represents an inheritance scheme or lineage.
• ancestorId Repository ID that represents an inheritance scheme or lineage.
This property is set to the repository ID of the item whose descendants you want to search, typically obtained through a hidden input tag, or supplied by the form user through an
input field.

D) Advanced Search:
• doAdvancedSearch If set to true.
• advancedSearchPropertyNames List of properties that are searched, where each property is
specified in this format:
item-type.property-name
• advancedSearchPropertyRanges Property range used to narrow search results by integer
values. The specified range is inclusive and uses this format:
item-type.property-name.max
item-type.property-name.min
You can leave the search range open at either end by
leaving the maximum or minimum value undefined.
• advancedSearchPropertyValues List of property values used in the search query. The format
should appear as follows:
item-type.property-name=value
• clearQueryURL The URL of the page to display when the handleClearQuery() method is invoked. The specified page should have search tools. If this property is empty,
• handleClearQuery() returns control to the current page.
• displayName The name displayed for an item type when it is used as search criteria.
• propertyValuesByType Holds the properties in advancedSearchPropertyNames
and their values in a HashMap key/value relationship. This only applies to those properties in
advancedSearchPropertyNames that are enumerated, RepositoryItems, or collection of RepositoryItems.
When the key is the name of a property, the value is a collection of possible values.
When the key is a repositoryId, the value is a collection of values for a specific property.
When the key is a collection of repositoryIds, the value is repositoryIds.
The format should appear as follows:
item-type.property-name

** Search Results Properties:
SearchFormHandler defines the following search results properties:
-currentResultPageNum : page number of active page. Set this property in order to display a given page. The default setting is 1.
-enableCountQuery Enables access to other query result properties.
-endCount : number of the last item on the current page.
-endIndex : based index number of the last item on the current page. The form handler uses this property to calculate the endCount property.
-maxResultsPerPage : Maximum number of items that display on a page.
-maxRowCount : Maximum number of items returned from a search. A value of –1 (the
default) returns all items that satisfy the search criteria.
-resultPageCount : Number of pages that hold search results.
-resultSetSize : Number of items returned from the search.
-startCount : number of the first item on the current page.
-startIndex : index number of the first item on the current page. The form handler uses this property to calculate the startCount property.

You can use these properties to construct the search results page.

12) DSP Tag List:
-dsp:a
Supports passing values to component properties and page parameters on click-through. Also handles URL rewriting.
-dsp:beginTransaction
Starts a transaction.
-dsp:commitTransaction
Completes a transaction.
-dsp:contains
Determines whether a Collection contains a specific single valued item.
-dsp:containsEJB
Determines whether a Collection contains an EJB.
-dsp:demarcateTransaction
Manages a transaction by starting the transaction, checking for errors, rolling back the transaction when errors are found and committing it when they are not.
-dsp:droplet
Invokes an ATG Servlet Bean.
-dsp:equalEJB
Determines whether two EJBs have the same primary keys.
-dsp:form Encloses a form that can send DSP form events.
-dsp:frame Embeds a page by encoding the frame src URL.
-dsp:getvalueof
Introduces a page parameter or component property value as an element in a JSP.
-dsp:go Encloses a form that can send WML form events.
-dsp:iframe Embeds a dynamic page, by encoding the frame src URL.
-dsp:img Inserts an image.
-dsp:importbean
Imports a Nucleus component into a page so it can be referred to without using its entire pathname. Also, creates a reference to a Nucleus component in an attribute visible to EL expressions.
-dsp:include
Embeds a page into another page.
-dsp:input
Passes values to a component property on submission.
-dsp:link
References a page, such as a stylesheet, by encoding the link src URLs.
-dsp:oparam
Specifies content to be rendered by an enclosing
-dsp:droplet tag.
-dsp:option
Specifies content to be rendered by an enclosing
-dsp:select tag.
-dsp:orderBy
Provides the sorting pattern to parent tag dsp:sort.
-dsp:page
Enables ATG page processing functionality.
-dsp:param
Stores a value in a parameter.
-dsp:postfield
Passes values to a component property on submission (WML).
-dsp:property
Sets a component property from dsp:a tag.
-dsp:rollbackTransaction
Causes any actions in the current transaction to be returned to their pretransaction state.
-dsp:select
Passes values to a component property on submission.
-dsp:setTransactionRollbackOnly
Specifies that, when a transaction is prompted to end, it ends in a rollback action.
-dsp:setvalue
Sets the value of a component property or a page parameter to a specified value.
-dsp:setxml Sets attribute display format to XML or HTML.
-dsp:sort
Organizes the contents of a Container or array based on a sorting pattern.
-dsp:test
Makes an object’s descriptive information available so other tags can find out its size, data type, and so on.
-dsp:textarea
Passes values to a component property on submission.
-dsp:tomap
Introduces a page parameter, standard JavaBean or Dynamic Bean component, or constant value as an element in a JSP that other tags can render using EL.
-dsp:transactionStatus
Reads the current transaction’s status.
-dsp:valueof
Retrieves and displays the value of a page parameter,component property, or constant value.

Reference: ATG Platform documentation set : Version 9.1 - 7/31/09

Monday, September 12, 2011

ATG Made Easy - part 3

9) JMS & ATG:

The Dynamo Application Framework includes a number of JMS-related tools, which are known collectively as the Dynamo Messaging System (DMS). The main parts of DMS are:

a) Two JMS providers, Local JMS and SQL JMS. Local JMS is built for high-speed lowlatency
synchronous messaging within a single process. SQL JMS is more robust, and
uses an SQL database to handle communication between components within the
same Dynamo application, or components running in different processes.

b) Patch Bay is an API and configuration system layered on top of JMS. Patch Bay is
designed to ease the development of messaging applications in Dynamo. The Patch
Bay API allows Nucleus components to send and receive messages. The configuration
system uses an XML file to specify how these components should be connected. This
file allows developers to change or add connections between components without
changing code.

**) Local JMS:

Local JMS does no queuing. When a message is sent, Local JMS immediately finds out who the receivers are and calls the appropriate methods on the receivers to deliver the message, waiting for each receiver to process the message before delivering the message to the next receiver. Only when the message has been delivered to all receivers does control return to the sender. In this way, Local JMS works more like Java Bean events than like typical JMS implementations; when a Java Bean fires an event, it actually calls a method on several registered listeners.
Local JMS is also non-durable; all messages are non-persistent. If a message is sent to a queue destination that has no listeners, the message disappears. Also, durable subscriptions to topic destinations act exactly like non-durable subscriptions—if a subscriber is not listening to a topic, it misses any messages sent to that topic whether it is subscribed durably or not.
Local JMS is most often used to pass data around to various components within a single request.

b) SQL JMS:
Asynchronous messaging and message persistence, it uses an SQL database for persistence of messages.
This ensures that messages are not lost in the event of system failure, and enables support for persistent queues and durable subscriptions, as described in Message Persistence.

c) Patch Bay:
Patch Bay is designed to simplify the process of creating JMS applications. Patch Bay includes a simplified API for creating Nucleus components that send and receive messages, and a configuration file where you declare these components and your JMS destinations. When a Nucleus-based application starts up, it examines this file and automatically creates the destinations and initializes the messaging components.
This means your code does not need to handle most of the JMS initialization tasks, such as obtaining a ConnectionFactory, obtaining a JMS Connection, and creating a JMS Session

Patch Bay is represented in Nucleus as the component /atg/dynamo/messaging/MessagingManager, which is of class atg.dms.patchbay.PatchBayManager

MessagingManager uses an XML file called the DMS configuration file to configure the individual parts of the Patch Pay system, such as JMS providers, message sources and sinks, and destinations. The definitionFile property of the MessagingManager component names the DMS configuration file.

Components: (all must be global scoped)
*Message source: A component that can send messages. A message source must implement the atg.dms.patchbay.MessageSource interface.
==>create and send messages
* Message sink: a component that can receive messages. A message sink must implement the atg.dms.patchbay.MessageSink interface.
==> This interface defines a single method, receiveMessage, which is called to notify the message sink that a message is being delivered.
* Message filter: a component that implements both interfaces, and can send and receive messages.
==>Message filters must implement both the MessageSource and the MessageSink interface. A message filter typically implements receiveMessage by manipulating the message in some way, then sending a new message.
* In addition to your sources and sinks, you must also define standard JMS destinations;

Patch Bay is represented in Nucleus as the component /atg/dynamo/messaging/MessagingManager.
The definitionFile property of the component MessagingManager names the XML file that
configures Patch Bay. The value of this property is:
/atg/dynamo/messaging/dynamoMessagingSystem.xml

Example for defining the 3 elements:
<?xml version="1.0" ?>
<dynamo-message-system>
<patchbay>
<message-source>
<nucleus-name>
/atg/dynamo/messaging/TestSource1
</nucleus-name>
</message-source>
<message-sink>
<nucleus-name>
/atg/dynamo/messaging/TestSink1
</nucleus-name>
</message-sink>
<message-filter>
<nucleus-name>
/atg/dynamo/messaging/TestFilter1
</nucleus-name>
</message-filter>
</patchbay>
</dynamo-message-system>

You can use SQL JMS as your JMS provider for your own applications. However, if you are running the ATG platform on IBM WebSphere Application Server or Oracle WebLogic Server, you might prefer to use your application server’s JMS provider.

NB: Use topic if you have multiple subscribers each provide different functionality and you want them all to list to it, and use Queue if you have only 1 functionality and you need only 1 type of listeners that process the messages only once.

10) Search Engine Optimization:
Search Engine Optimization (SEO) is a term used to describe a variety of techniques for making pages more accessible to web spiders (also known as web crawlers or robots), the scripts used by Internet search engines to crawl the Web to gather pages for indexing. The goal of SEO is to increase the ranking of the indexed pages in search results.

Available Options:
-URL Recoding
-Canonical URLs
-Sitemaps
-SEO Tagging

*Jumping Servlet:
The atg.repository.seo.JumpServlet class is responsible for translating static request URLs to their dynamic equivalents. This class extends the atg.servlet.pipeline.InsertableServletImpl class, so it can be inserted in the DAS or DAF servlet pipeline. However, because this servlet is intended to process only static URLs, and incoming URLs are typically dynamic, including the servlet in a pipeline may be very inefficient. Therefore, it is generally preferable to configure it as a URI-mapped servlet in the web.xml file of your application, to ensure that it processes only static URLs.

To configure the jump servlet in a web.xml file, you actually declare another class,
atg.repository.seo.MappedJumpServlet. This is a helper class that invokes the JumpServlet
component. In addition, you declare a servlet mapping for the pattern that the servlet uses to detect static request URLs.
For example, if you have configured your static URLs to include /jump/ immediately after the context root, the entry in the web.xml file would be something like this:
<servlet>
<servlet-name>MappedJumpServlet</servlet-name>
<servlet-class>atg.repository.seo.MappedJumpServlet</servlet-class>
<init-param>
<param-name>jumpServlet</param-name>
<param-value>ctx:dynamo:/atg/repository/seo/JumpServlet</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>MappedJumpServlet</servlet-name>
<url-pattern>/jump/*</url-pattern>
</servlet-mapping>

There also are several properties you can configure for the Nucleus component:
- templates
An array of IndirectUrlTemplate components that the servlet examines in the order specified until it finds one that matches the static request URL.
- defaultRepository
Specifies the repository to associate with repository items for which a repository is not
otherwise specified.
- defaultWebApp
Specifies the default web application to use when determining the context path for a URL.

In addition, the servlet has nextServlet and insertAfterServlet properties for including the
component in a servlet pipeline. If the servlet is configured through the web.xml file, you should not set these properties.

The templates point to a global component property files for the component: IndirectUrlTemplate or DirectUrlTemplate (atg.repository.seo.InDirectUrlTemplate and atg.repository.seo.DirectUrlTemplate) where you can specify many properties including:

indirectRegex=/shop/(mobile-phone)/(.*)/(.*)$
regexElementList=parentAlias | string,\
skuItem | id | /atg/commerce/catalog/ProductCatalog:skuLookup,sTab | string
urlTemplateFormat=/shop/{parentAlias}/{skuItem.id}/{sTab}

Then the forwarded URL by the jump servlet like:

forwardUrlTemplateFormat=/{skuItem.sku.template.URL,encode=true}?skuId\={skuItem.sku.id}&selectedTab\={sTab}

Another example: suppose you have a URL format that looks like this:

urlTemplateFormat=/jump/product/{item.id}/{item.parentCategory.id}\
/{item.displayName}/{item.parentCategory.displayName}

The regular expression pattern for this format might be specified like this:
indirectRegex=/jump/product/([^/].*?)/([^/].*?)/([^/].*?)/([^/].*?)$

Reference: ATG Platform documentation set : Version 9.1 - 7/31/09