Saturday, October 16, 2010

High CPU utilization of JAVA process...what is going on?

On top of one kernel space thread LWP is running and user space threads are running on top of LWP.
It may be 1:1 or one: many. I just tried out finding thread ID (LWPID) of java process. The CPU utilization of java PID was too high. Usually I use following command
ps -Leo pid,ruser,vsz,rss,state,priority,nice,time,%cpu,comm,lwp,psr,nlwp
pid = Process ID
ruser = Real user ID
vsz = virtual memory size of the process in KiB (1024-byte units).
rss = resident set size, the non-swapped physical memory that a task has used (in kiloBytes).
state = Process state
priority = kernel scheduling priority.
nice = Nice value.
time = cumulative CPU time,
%cpu = cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage. It will not add up to 100% unless you are lucky. (alias pcpu).
comm = command name (only the executable name). Modifications to the command name will not be shown. A process marked is partly dead, waiting to be fully destroyed by its parent. The output in this column may contain spaces. (alias ucmd, ucomm). See also the args format keyword, the -f option, and the c option. When specified last, this column will extend to the edge of the display. If ps can not determine display width, as when output is redirected (piped) into a file or another command, the output width is undefined. (it may be 80, unlimited, determined by the TERM variable, and so on) The COLUMNS environment variable or --cols option may be used to exactly determine the width in this case. The w or -w option may be also be used to adjust width.
lwp and nlwp = lwp (light weight process, or thread) ID of the lwp being reported and number of lwp.
psr = processor that process is currently assigned to.
I have just kept it here for my own convenience. use man to get help of command.
As it was Weblogic server and java processes were owned by weblogic user, I used “| grep weblogic” and then filter with “ grep -v ' 0.0 '” to minimize output.
The above command gives us thread ID and CPU utilization of each thread. Many ways we can accomplish this task.
Use “ps –eo user,pid,%cpu,cmd | grep java” and then “ps u –Lp ” or “top –H –p .
Now we have to search thread id in process thread dump. Before that create thread dump using “kill -3 ” or JVM specific command (JDK/Jrokit).
#jstack > threadDump.txt
OR
#jrcmd print_threads > threadDump.txt
Search lwp ID as tid in thread dump that has taken from Jrokit JVM. Use hex value of lwp ID if java process uses jdk.

No comments:

Post a Comment