Websphere application server (WAS) – Thread dump and Heap Dump


When to generate ? How to generate ? how to debug ?

Thread Dumps
If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a thread dump to help diagnose the problem.

In the case of a server hang, you can force an application to create a thread dump.

On unix/Linux machines find the process id (PID) of the hung JVM and issue kill -3 PID.  Look for an output file in the installation root directory with a name like javacore.date.time.id.txt.
Using wasadmin prompt,
get the handle of the server
wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
execute
wsadmin>$AdminControl invoke $jvm dumpThreads Continue reading

JVM Tools – JPS


JPS

The jps tool lists the instrumented HotSpot Java Virtual Machines (JVMs) on the target system. The tool is limited to reporting information on JVMs for which it has the access permissions.If jps is run without specifying a hostid, it will look for instrumented JVMs on the local host.

Usage:jps [ options ] [ hostid ]

The host identifier of the host for which the process report should be generated The general options include:

# q -Suppress the output of the class name, JAR file name, and arguments passed to the main method, producing only a list of local VM identifiers.
# m -Output the arguments passed to the main method. The output may be null for embedded JVMs.
# l -Output the full package name for the application’s main class or the full path name to the application’s JAR file.
# v -Output the arguments passed to the JVM.
# V -Output the arguments passed to the JVM through the flags file (the
.hotspotrc file or the file specified by the -XX:Flags=<filename> argument).
# Joption -Pass option to the java launcher called by javac. For example, -J-Xms48m sets the startup memory to 48 megabytes Continue reading

JVM Tools – JMAP


JMAP

The Java jmap utility provides a number of useful options to summarize heap usage, and get a break down of objects in the new and old generations. To summarize the new and old generations, the jmap utility can be run with the “-heap” option, and the PID of the JVM to summarize (the PID can be acquired by running the jps utility, or by reviewing the ps output):

Usage: jmap [ option ] pid
jmap [ option ] executable core

General Options include:

# jmap -heap prints heap usage, gc algorithm and heap configuration details.
# jmap -histo prints a histogram of the current heap or core (class, number of instances, and the total number of bytes used by all the instances together).
# jmap -permstat prints details of the perm space contents.
# jmap -permgen prints statistics for the objects in the permanent generation, including information about internalized String instances. Continue reading

JVM Tools – JSTAT


JSTAT:

JSTAT is Java Virtual Machine Statistics Monitoring Tool. jstat provides information on performance and resource consumption of running applications.

Usage: jstat [ outputOptions vmid [interval[s|ms] [count]] ]

If you do not specify a general option, then you can specify output options. Output options determine the content and format of jstat’s output, and consist of a single statOption, plus any of the other output options ( -h, -t, and -J). The statOption must come first. Continue reading

DCS and Cluster members


Have you ever been asked this question in the interview?

how do you find out which cluster member was crashed/down?

The general answer we give is to go to administration console and check the individual server status or the cluster member status.

The other option is to use a third-party monitoring tool such as ITCAM, wily introscope, UniCenter and Nagios etc..

Have you ever checked the system.out log file of any individual server when one of the cluster member was stopped?

WebSphere has Distribution & Consistency Services (DCS), which is a part of the HA architecture. Using these DCS messages we can find which member of the cluster is down.

Here is an example: Continue reading

Class Loader & ClassNotFound exception in WebSphere


A class loader is an object that is responsible for loading classes. The class ClassLoader is an abstract class. Given the name of a class, a class loader should attempt to locate or generate data that constitutes a definition for the class. A typical strategy is to transform the name into a file name and then read a class file of that name from a file system.

When a class loading request is presented to a class loader, it first asks its parent class loader to fulfill the request. The parent class loader, in turn, asks its parent for the class until the request reaches the top of the hierarchy. If the class loader at the top of the hierarchy cannot fulfill the request to load a class, then the child class loader that called it is responsible for loading the class. If the child is also unable to load the class, the request continues back down the hierarchy until a class loader fulfils it or a ClassNotFoundException is produced by the last class loader.

image

 

Each class loader is a child of the previous class loader. That is, the application module class loaders are children of the WebSphere extensions class loader, which is a child of the CLASSPATH Java class loader. Design and packaging of an application will determine the behavior of class loading. WebSphere provides the ability to change/modify the class loading behavior.

WebSphere Class Loaders:

WebSphere application server has 3 class loaders.

  • Application server class loader
    The application server class loader policy affects all applications that are deployed on the server.
  • Enterprise application class loader
    An application class loader is the parent class of an Enterprise application (EAR) and all modules within it. An application class loader groups enterprise bean (EJB) modules, shared libraries, and dependency Java archive (JAR) files associated to an application. Dependency JAR files are JAR files that contain code which can be used by both enterprise beans and servlets.
  • Web module class loader
    A web module has its own Web application archive (WAR) class loader to load the contents of the web module, which are in the WEB-INF/classes and WEB-INF/lib directories.

1. Application server class loader

Go to Servers –> Application Servers –> Server name

image

Look for the above options.

Classloader Policy

Single: Applications are not isolated from each other. Uses a single application class loader to load all of the EJB modules, shared libraries, and JAR files which are contained in all applications installed into the JVM.
Multiple: Applications are isolated from each other. Gives each application its own class loader to load the EJB modules, shared libraries, and JAR files.

Class loading mode

parent first: Sets the loading of classes to its parent class loader before attempting to load the class from its local class path. This is the default value for Class loading mode

parent last: Tells the class loader to start with loading classes from its local class path before asking its parent.

For each application server in the system, you can set the application class-loader policy to Single or Multiple. When the application class-loader policy is set to Single, then a single application class loader loads all EJB modules, dependency JAR files, and shared libraries in the system. When the application class-loader policy is set to Multiple, then each application receives its own class loader that is used for loading the EJB modules, dependency JAR files, and shared libraries for that application.

Note: If you’ve multiple application running on the same server (JVM) and if their classes are conflicting each other or Also some times it can happen that, application classes may conflict with the WebSphere classes. then we change the class loading mode option from default.

2. Application Class Loader

In general Enterprise applications (EAR) will have multiple web, ejbs or sometimes may include application client modules. Enterprise applications can also override settings within the contained
modules deployment descriptors to combine or deploy them. By placing JAR files in the enterprise application instead of the global class path of an application server, they are also within the application and thus they get deployed along with the application. The concept is that an EAR file encapsulates all its required resources and hence it can be pre-configured using some java techniques.

Go to Applications –> Enterprise applications –> Application name –> Class loading and updation

image

you can see the above options

Class loader order

Classes loaded with parent class loader first
Sets the loading of classes to its parent class loader before attempting to load the class from its local class path.
Classes loaded with application class loader first
Tells the class loader to start with loading classes from its local class path before asking its parent

WAR class loader policy

Class loader for each WAR file in application
A separate class loader is assigned to each WAR file.
Single class loader for application
One class loader is assigned to all WAR files.

An application class loader loads classes from Web modules if the application’s WAR class-loader policy is set to Application. If the application’s WAR class-loader policy is set to Module, then each WAR module receives its own class loader.

3. Web module class loader

Every web module will have 2 folders, WEB-INF/classes and WEB-INF/lib. The classes folder may contain Java classes within the web application. Then we can specify a class loader to looks at this folder to load those classes.  Remember these classes are only for that specific web module.

image

Class loader order

Specifies whether the class loader searches in the parent class loader or in the application class loader first to load a class. The standard for development kit class loaders and product class loaders is Classes loaded with parent class loader first. By specifying Classes loaded with application class loader first, your application can override classes contained in the parent class loader, but this action can potentially result in ClassCastException or LinkageErrors if you have mixed use of overridden classes and non-overridden classes.

Classes loaded with parent class loader first
If set, the class loader searches the application’s class loader for the class.
Classes loaded with application class loader first
If set, the class loader searches within the WAR class loader first to load a class.

Class-loader isolation

The number and function of the application module class loaders depend on the class-loader policies that are specified in the server configuration. Class loaders provide multiple options for isolating applications and modules to enable different application packaging schemes to run on an application server

Class loader isolation combinations

Type of Isolation Application server Class loader WAR class loader
Full Multiple WAR
Partial Multiple Application
Minimum Single Application

you might be wondering, where is the 4th option (Application server class loader – Single + WAR class loader WAR). Once you set the application class loader to Single, there is only one class loader for entire server. That’s why this option was not mentioned :-)

Tip: check the below link to learn about these policies with an example

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/crun_classload.html

WebSphere Application Server Troubleshooting tool : IBM Support Assistant v5


IBM Support Assistant (ISA) 5.0 Tech Preview introduces the next generation of IBM’s popular problem determination and troubleshooting platform. This new, complimentary release provides the richest tools and functions you need to perform root-cause analysis and enhance your productivity.

New features:

  • Case Management – simplify the organization of your problem determination investigation
  • File Management – enhanced capabilities to help you search through files quickly
  • Tools – automated analysis tools to generate reports and assist you in problem discovery
  • Intuitive web UI – with an emphasis on simplicity, you can easily open a browser to access the ISA application
  • Server-based application – install ISA once and collaborate with multiple team members. Analysis processing can now be off-loaded from your desktop!
  • Single-user desktop – run the ISA application on your desktop if you prefer a single-user mode