0

Large number of threads (java.lang.Object.wait)

asked 2011-03-31 20:44:38 +0800

estuardoumg gravatar image estuardoumg
15 1

Hi,

We are getting a lot of messages of hung Threads in many applications developed with ZK framework. The first time we saw the error "... may be hung" in the SystemOut.log file we thought was the event thread model, so we made a change in the zk.xml and added the following configuration

(WebContainer : 4428" (000011c3) has been active for 66757 milliseconds and may be hung)

<system-config>
<disable-event-thread>false</disable-event-thread>
</system-config>

Once we made the change we put againt the application in the production enviroment but we got the same errors in more recurrent than before.

The ZK version in our applications is ZK 5.0.5 and this applications has between 500 and 600 users.

The applications are running under Websphere Application Server 6.1x and under SunOS 5.10, we analyzed the thread dump hoping to find some errors in our classes but we didn't find anything.

All the stacktrace in the thread dump are like this:

Stacktrace 1

at java.lang.Object.wait(Native Method) - waiting on [0xfffffffe8267b180] (a java.lang.Object) at org.zkoss.zk.ui.impl.UiEngineImpl.doActivate(UiEngineImpl.java:1687) - locked [0xfffffffe8267b180] (a java.lang.Object) at org.zkoss.zk.ui.impl.UiEngineImpl.activate(UiEngineImpl.java:1636) at org.zkoss.zk.ui.impl.UiEngineImpl.desktopDestroyed(UiEngineImpl.java:192) at org.zkoss.zk.ui.impl.SimpleDesktopCache.desktopDestroyed(SimpleDesktopCache.java:121) at org.zkoss.zk.ui.impl.SimpleDesktopCache.stop(SimpleDesktopCache.java:178) - locked [0xfffffffe8225d488] (a org.zkoss.zk.ui.impl.SimpleDesktopCache$Cache) at org.zkoss.zk.ui.impl.SessionDesktopCacheProvider.sessionDestroyed(SessionDesktopCacheProvider.java:59) at org.zkoss.zk.ui.impl.AbstractWebApp.sessionDestroyed(AbstractWebApp.java:321) at org.zkoss.zk.ui.http.WebManager.sessionDestroyed(WebManager.java:386) at org.zkoss.zk.ui.http.HttpSessionListener23.sessionDestroyed(HttpSessionListener23.java:45) at com.ibm.ws.webcontainer.httpsession.SessionContext.sessionDestroyedEvent(SessionContext.java:3559) at com.ibm.ws.webcontainer.httpsession.SessionData.sessionUnbound(SessionData.java:1018) at com.ibm.ws.webcontainer.httpsession.SessionContext.notifySessionInvalidated(SessionContext.java:1094) at com.ibm.ws.webcontainer.httpsession.SessionData.invalidate(SessionData.java:621) - locked [0xfffffffe81e4dff8] (a com.ibm.ws.webcontainer.httpsession.MemorySessionData) at com.ibm.ws.webcontainer.httpsession.HttpSessionFacade.invalidate(HttpSessionFacade.java:181) at org.zkoss.zk.ui.http.SimpleSession.invalidateNow(SimpleSession.java:346) at org.zkoss.zk.ui.impl.UiEngineImpl.doDeactivate(UiEngineImpl.java:1776) at org.zkoss.zk.ui.impl.UiEngineImpl.execUpdate(UiEngineImpl.java:1170) at org.zkoss.zk.au.http.DHtmlUpdateServlet.process(DHtmlUpdateServlet.java:561) at org.zkoss.zk.au.http.DHtmlUpdateServlet.doGet(DHtmlUpdateServlet.java:456) at org.zkoss.zk.au.http.DHtmlUpdateServlet.doPost(DHtmlUpdateServlet.java:464) at javax.servlet.http.HttpServlet.service(HttpServlet.java:763) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:276) at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:463) at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:281) at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:256) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:989) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:501) at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:464) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:113) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:383) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:102) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:195) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:873) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1469)

Stacktrace 2

at com.ibm.ws.webcontainer.httpsession.SessionData.getValueGuts(SessionData.java:836) - waiting to lock [0xffffffff57d24690] (a com.ibm.ws.webcontainer.httpsession.MemorySessionData) at com.ibm.ws.webcontainer.httpsession.SessionData.getValue(SessionData.java:1243) at com.ibm.ws.webcontainer.httpsession.SessionData.getAttribute(SessionData.java:1129) at org.zkoss.zk.ui.http.SimpleSession.getAttribute(SimpleSession.java:218) at org.zkoss.zk.ui.http.SimpleSession.hasAttribute(SimpleSession.java:226) at org.zkoss.zk.ui.http.SimpleSession.hasAttribute(SimpleSession.java:256) at org.zkoss.zk.ui.impl.DesktopImpl.hasAttribute(DesktopImpl.java:587) at org.zkoss.zk.ui.impl.PageImpl.hasAttributeOrFellow(PageImpl.java:389) at org.zkoss.zk.ui.AbstractComponent.hasAttributeOrFellow(AbstractComponent.java:918) at org.zkoss.zk.ui.Components$Wire.containsVariable(Components.java:1068) at org.zkoss.zk.ui.Components$Wire.wireOthers(Components.java:1034) at org.zkoss.zk.ui.Components$Wire.myWireVariables(Components.java:975) at org.zkoss.zk.ui.Components$Wire.wireVariables(Components.java:971) at org.zkoss.zk.ui.Components$Wire.access$500(Components.java:880) at org.zkoss.zk.ui.Components.wireVariables(Components.java:525) at org.zkoss.zk.ui.util.GenericAutowireComposer$BeforeCreateWireListener.onEvent(GenericAutowireComposer.java:232) at org.zkoss.zk.ui.impl.EventProcessor.process0(EventProcessor.java:177) at org.zkoss.zk.ui.impl.EventProcessor.process(EventProcessor.java:140) at org.zkoss.zk.ui.impl.UiEngineImpl.processEvent(UiEngineImpl.java:1602) at org.zkoss.zk.ui.impl.UiEngineImpl.process(UiEngineImpl.java:1386) at org.zkoss.zk.ui.impl.UiEngineImpl.execNewPage0(UiEngineImpl.java:446) at org.zkoss.zk.ui.impl.UiEngineImpl.execNewPage(UiEngineImpl.java:305) at org.zkoss.zk.ui.http.DHtmlLayoutServlet.process(DHtmlLayoutServlet.java:225) at org.zkoss.zk.ui.http.DHtmlLayoutServlet.doGet(DHtmlLayoutServlet.java:146) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:276) at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:463) at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:281) at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:256) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:989) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:501) at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:464) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:113) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:383) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:263) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:195) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:873) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1469)


Thanks in advance for your help

Greetings,

delete flag offensive retag edit

9 Replies

Sort by ยป oldest newest

answered 2011-03-31 23:26:50 +0800

samchuang gravatar image samchuang
4084 4

updated 2011-03-31 23:27:06 +0800

Hi

1.
Could you try to disable thread ? using ZK's default. Please refer to Event Threads

It mentioned
"By default, ZK processes an event in the same Servlet thread that receives the HTTP request. It is the suggested approach because the performance is better and it is easy to integrate with other framework"

Which means

<system-config>
<disable-event-thread>true</disable-event-thread>
</system-config>

2. Do you use server push ? It may also cause the issue. If you use server push, how do you use it ?

link publish delete flag offensive edit

answered 2011-04-01 01:22:07 +0800

estuardoumg gravatar image estuardoumg
15 1

Hi samchuang, thanks for your replay.

1. Actually im using the default configuration of ZK 5, i understood zk 5 has the thread event disabled. And for example when a call a modal window or show a message box the thread is not suspended, for that reasson i had to implement code like described in the link http://books.zkoss.org/wiki/ZK_Developer%27s_Reference/UI_Patterns/Event_Threads/Messagebox.

I made a mistake writing <disable-event-thread>false</disable-event-thread>, my zk.xml looks like this

<zk>
<device-config>
<device-type>ajax</device-type>
<timeout-uri>/index.zul</timeout-uri><!-- An empty URL can cause the browser to reload the same URL -->
</device-config>
<session-config>
<session-timeout>1800</session-timeout>
<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>
</session-config>
<!-- Para las pruebas de rendimiento con selenium
<system-config>
<id-generator-class>com.consystec.selenium.idgenerator.ConsystecIdGenerator</id-generator-class>
</system-config> -->
</zk>

2. I don't use server push


I also noticed (Tivoli Performance - WAS) when the application has a low demand works fine (lower than 50 servlet sessions), when the number of sessions grows, begins the hanging threads immediately. Whith the event thread enable it occurs too but occasionally, so I decided to disable the event thread to minimize the incidence but the opposite happened. May be its because the amount of the application users and sessions, but i do not know what else to do.

The application its very simple, has 4 pages and in the last page redirects to others sites (http://www.youtube.com/watch?v=BEJ9HQT1vt8)

Thank you again in advance.

link publish delete flag offensive edit

answered 2011-04-05 17:16:47 +0800

estuardoumg gravatar image estuardoumg
15 1

Anyone has the same problem with too many users ??

link publish delete flag offensive edit

answered 2011-04-05 19:55:22 +0800

samchuang gravatar image samchuang
4084 4

updated 2011-04-05 19:59:43 +0800

Hi

is't possible to provide a War file that can reproduce the issue ?

by the way, just for curious, do you have any reason for these ?

<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>

link publish delete flag offensive edit

answered 2011-05-27 15:54:08 +0800

bfontana gravatar image bfontana
51 1

We also had the same problem under heavy load

link publish delete flag offensive edit

answered 2011-05-27 17:28:42 +0800

estuardoumg gravatar image estuardoumg
15 1

updated 2011-05-27 19:14:22 +0800

Something I saw, when the application has the event thread enable the application works fine for a week,
then starts the hung threads. When i changed to event thread disable, the thread pool of websphere was
full in less than 30 min.

This week i'm going to deploy a new version of this application with ZK 5.0.7 EE with event thread enabled
and see what happends in production enviroment. I do not want to ask to the server administrator any load balancing, because it involves
purchasing a license from IBM and maybe serialize everything in my application.

bfontana, what application server are you using ?? Could be websphere application server or could be ZK framework, i'm
really worry about.

samchuang, the reason for these:

<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>

was just testing, my zk.xml just have the default tags and the disable-event-thread tag.

And in the other hand samchuang, i ask to my superior if we can provide the source code but he doesn't let me.

link publish delete flag offensive edit

answered 2011-05-28 10:38:08 +0800

bfontana gravatar image bfontana
51 1

We are using:

Tomcat 5.5.25
Java 1.6 64 bits Sun's JVM (I don't remember update but I could find it if it helps).
RedHat Linux Enteprise Server
"physical" server is a virtualized VMWare VM with 4 Xeon cores assigned and 6 GB RAM.

So I'm afraid is not a WebSphere/ZK issue.

link publish delete flag offensive edit

answered 2011-05-28 11:48:20 +0800

bfontana gravatar image bfontana
51 1

Oh I forgot. We are using ZK 5.0.4 and ZKSpring 3.0-FL-2010-04-20

link publish delete flag offensive edit

answered 2011-10-11 16:20:15 +0800

estuardoumg gravatar image estuardoumg
15 1

Hi,

Good news, problem solved. I updated the version of zk to 5.0.7, and in the xml just disabled the event threads, taking into consideration all the changes discribed in this seccion and subsection http://books.zkoss.org/wiki/ZK%20Developer's%20Reference/UI%20Patterns/Event%20Threads

Also increase the maximum threads in webcontainer to improve the performance as seen in the Tivoli Performance Viewer.

link publish delete flag offensive edit
Your reply
Please start posting your answer anonymously - your answer will be saved within the current session and published after you log in or create a new account. Please try to give a substantial answer, for discussions, please use comments and please do remember to vote (after you log in)!

[hide preview]

Question tools

Follow

RSS

Stats

Asked: 2011-03-31 20:44:38 +0800

Seen: 836 times

Last updated: Oct 11 '11

Support Options
  • Email Support
  • Training
  • Consulting
  • Outsourcing
Learn More