WebSphere Application Server Troubleshooting tool : IBM Support Assistant v5

IBM Support Assistant (ISA) 5.0 Tech Preview introduces the next generation of IBM’s popular problem determination and troubleshooting platform. This new, complimentary release provides the richest tools and functions you need to perform root-cause analysis and enhance your productivity.

New features:

  • Case Management – simplify the organization of your problem determination investigation
  • File Management – enhanced capabilities to help you search through files quickly
  • Tools – automated analysis tools to generate reports and assist you in problem discovery
  • Intuitive web UI – with an emphasis on simplicity, you can easily open a browser to access the ISA application
  • Server-based application – install ISA once and collaborate with multiple team members. Analysis processing can now be off-loaded from your desktop!
  • Single-user desktop – run the ISA application on your desktop if you prefer a single-user mode

Hung thread detection policy in WebSphere Application Server

The hung thread detection policy in websphere application server lets you detect the hung threads by sending notifications in the log files. You can configure a hang detection policy to, accommodate your applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. Using a hang thread detection policy you can specify a time that is too long for a unit of work to complete. A thread monitor will check all the threads in the systems.

Hang thread detection option is enabled by default. You can adjust the values of the detection policy or disable it by using the below procedure.

  • Navigate to Servers –> Applicaiton Servers –> server_name –> administration –> custom properties
  • Add the following 4 custom properties
# Property Description Default
A com.ibm.websphere.threadmonitor.interval  How frequently thread monitor should check all the managed threads for hung threads 180
B com.ibm.websphere.threadmonitor.threshold  After how many seconds a thread can be considered as hung 600
C com.ibm.websphere.threadmonitor.false.alarm.threshold The number of times that false alarms can occur before automatically increasing the threshold (T) 100
D com.ibm.websphere.threadmonitor.dump.java  when set to ‘true’, creates a java core when a hung thread is detected false
  • Click OK and save the changes.
  • Sync the changes and restart the servers for the changes to take effect.

False Alarms:

What happens if a thread eventually finishes its work after it been reported as hung? This situation is called a ‘false alarm’. A large number of these events indicate that the threshold value is too small. The hang detection facility can automatically respond to this situation. For every (T) false alarm, the threshold is increased by a factor of 1.5

Notifications for J2EE applications:

The notifications from the hand thread detector will appear in the log file.

Hung thread report  WSVR0605W Thread ‘thread_name’ has been active for ‘xxx sec’ and may be hung.  There are ‘N’ threads in total in the server that may be hung.
False Alarm  WSVR0606W Thread ‘thread_name’ was previously reported to be hung but has completed. It was active for approximately ‘yyy sec’. There are ‘N’ threads in total in the server that still may be hung.
Auto adjustment  WSVR0607W Too many thread hangs have been falsely reported.  The hang threshold is now being set to ‘zzz sec’.

WebSphere Application Server Troubleshooting, part-7 : Web Server

The following has been written based on the assumption that you are using either IBM http server or Apache in front of your websphere application server.

So… how to troubleshoot issues with web server ?

  • First and most important thing is to know, if the web server is really running. If you are on unix machines check for httpd process using ps command and also look for pid file under the logs directory. On windows machine you can check in the task manager for the process.
  • Next thing is log files. There are two log files in the web servers: access log and error log. Access log contains the logging of who accessed what on that web server. Error log contains the start-up messages and any error messages. It is important to note that, when you are using virtual-hosts in the web servers , each virtual host will have its own access and error logs.
  • Next important thing you need to remember are the HTTP response codes. These codes are explained here in detail. Until you get familiar with these response codes.. you can take a printout of this image and place it in your desk.
  • I generally receive emails from readers asking .. how do you troubleshot a 404 error or 500 error. A simple answer for 404 is, since 404 is page not found, check if the request url is correct.. if so then check if the content is available. similarly for 50x, it could be a configuration issue or your back-end server didn’t responded. in these kind of issues the above image will help you get an idea where to look for.
  • The other good idea is to enable the “server-status” option in the web server. But when you enable this option you need to keep in mind that, you need to restrict the access to this page. More details about server-status and how it can help can be found here.
  • Another useful tool is , apachectl command. Using this you can test a new configuration, check the syntax, temporarly change log level etc.. some of these are discussed in our Admintoolkit post, here.

While troubleshooting any issue… first make sure that the first component that is getting the requests is working fine. So before jumping onto websphere plugin and application servers… make sure web server is working as expected.

WebSphere Application Server Troubleshooting, part-5: Visual Configuration Explorer

Visual Configuration Explorer (VCE) is a visual tool for middleware configuration exploring, analysis, and problem diagnosis. VCE allows you to visualize your middleware configuration and easily search for specific property values, compare snapshots of different configurations and generate and share configuration analysis results.

VCE related forum on IBM developer works : http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1139

WebSphere Application Server Troubleshooting, part-4 : Garbage Collection and Memory Visualizer

The Garbage Collection and Memory Visualizer (GCMV) helps visualise the memory footprint, garbage collection behaviour and performance of Java Applications using verbose garbage collection output. The Garbage Collection and Memory Visualizer provides recommendations for tuning the application.

more info about GCMV : http://www.ibm.com/developerworks/java/jdk/tools/gcmv/