This process is an addition to the regular debugging techniques that include:
- Check all the processes and resources (CPU, RAM, IO, disk space, etc.)
- Check log files for errors
- Check network connectivity
- Check DNS (JVM by default caches the name resolution of hosts. So, if you have changed an IP for a certain name – you’ll need to restart the app to cache the new one)
Alright, these are the additions:
Java processes are usually multithreaded. So, if you just run “ps auxww | grep java” -> chances are that you’ll only see the parent processes.If you want to see all the threads and the resources they are taking – do this (replace JAVA_PID with the parent process ID number):
> top -Hp JAVA_PID
or if you don’t have “top” installed:
ps -C java -L -o pcpu,cpu,nice,state,cputime,pid,tid
Make a note of the highest CPU/memory consuming threads and their PIDs
Next – we need to get a thread dump of all the threads (without affecting the application or stopping the server) – $JAVA_HOME is where your java path is and JAVA_PID is the parent process ID number (same one from step 1); $USER is the name of the user the application is running as:
> sudo -u $USER $JAVA_HOME/bin/jstack JAVA_PID > threads.out
If everything went ok with step 3 – you’ll have a file with info about all the threads.
- You can examine the threads.out file and see which threads are BLOCKED or in “WAITING (on object monitor)” status. They will also tell you what LOCKED object they are waiting for. You can examine the file further to see exactly which thread is locking the object that all the blocked threads are waiting for. Then you can give the info to the developers and they should know how and what to fix (we hope).
All the threads IDs in thread.out file are in HEX format. Now, get the thread ID you took a note of in step 2 and convert it into HEX format (let’s say the highest CPU PID of the thread was 12345):
> printf “0x%xn” 12345
This will produce a result something like: 0×3039
Then you take the HEX ID (in this case 0×3039) -> and you track it down in the threads.out file to see info for that particular thread.
- If you don’t have a commercial java profiler (like Wiley) -> since JDK 1.6.7 there is a visual profiler included with your jdk
This is a guest post by Nick Stoianov. Nick is a System Administrator with expert experience configuring Apache, MySQL, and WebSphere. He is very adept at troubleshooting web applications as well as tuning web server environments for performance