# Large number of threads (java.lang.Object.wait)

estuardoumg
15 1

Hi,

We are getting a lot of messages of hung Threads in many applications developed with ZK framework. The first time we saw the error "... may be hung" in the SystemOut.log file we thought was the event thread model, so we made a change in the zk.xml and added the following configuration

(WebContainer : 4428" (000011c3) has been active for 66757 milliseconds and may be hung)

<system-config>
</system-config>

Once we made the change we put againt the application in the production enviroment but we got the same errors in more recurrent than before.

The ZK version in our applications is ZK 5.0.5 and this applications has between 500 and 600 users.

The applications are running under Websphere Application Server 6.1x and under SunOS 5.10, we analyzed the thread dump hoping to find some errors in our classes but we didn't find anything.

All the stacktrace in the thread dump are like this:

Stacktrace 1

at java.lang.Object.wait(Native Method) - waiting on [0xfffffffe8267b180] (a java.lang.Object) at org.zkoss.zk.ui.impl.UiEngineImpl.doActivate(UiEngineImpl.java:1687) - locked [0xfffffffe8267b180] (a java.lang.Object) at org.zkoss.zk.ui.impl.UiEngineImpl.activate(UiEngineImpl.java:1636) at org.zkoss.zk.ui.impl.UiEngineImpl.desktopDestroyed(UiEngineImpl.java:192) at org.zkoss.zk.ui.impl.SimpleDesktopCache.desktopDestroyed(SimpleDesktopCache.java:121) at org.zkoss.zk.ui.impl.SimpleDesktopCache.stop(SimpleDesktopCache.java:178) - locked [0xfffffffe8225d488] (a org.zkoss.zk.ui.impl.SimpleDesktopCache$Cache) at org.zkoss.zk.ui.impl.SessionDesktopCacheProvider.sessionDestroyed(SessionDesktopCacheProvider.java:59) at org.zkoss.zk.ui.impl.AbstractWebApp.sessionDestroyed(AbstractWebApp.java:321) at org.zkoss.zk.ui.http.WebManager.sessionDestroyed(WebManager.java:386) at org.zkoss.zk.ui.http.HttpSessionListener23.sessionDestroyed(HttpSessionListener23.java:45) at com.ibm.ws.webcontainer.httpsession.SessionContext.sessionDestroyedEvent(SessionContext.java:3559) at com.ibm.ws.webcontainer.httpsession.SessionData.sessionUnbound(SessionData.java:1018) at com.ibm.ws.webcontainer.httpsession.SessionContext.notifySessionInvalidated(SessionContext.java:1094) at com.ibm.ws.webcontainer.httpsession.SessionData.invalidate(SessionData.java:621) - locked [0xfffffffe81e4dff8] (a com.ibm.ws.webcontainer.httpsession.MemorySessionData) at com.ibm.ws.webcontainer.httpsession.HttpSessionFacade.invalidate(HttpSessionFacade.java:181) at org.zkoss.zk.ui.http.SimpleSession.invalidateNow(SimpleSession.java:346) at org.zkoss.zk.ui.impl.UiEngineImpl.doDeactivate(UiEngineImpl.java:1776) at org.zkoss.zk.ui.impl.UiEngineImpl.execUpdate(UiEngineImpl.java:1170) at org.zkoss.zk.au.http.DHtmlUpdateServlet.process(DHtmlUpdateServlet.java:561) at org.zkoss.zk.au.http.DHtmlUpdateServlet.doGet(DHtmlUpdateServlet.java:456) at org.zkoss.zk.au.http.DHtmlUpdateServlet.doPost(DHtmlUpdateServlet.java:464) at javax.servlet.http.HttpServlet.service(HttpServlet.java:763) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:276) at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:463) at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:281) at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:256) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:989) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:501) at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:464) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:113) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:383) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:102) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:195) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:873) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1469) Stacktrace 2 at com.ibm.ws.webcontainer.httpsession.SessionData.getValueGuts(SessionData.java:836) - waiting to lock [0xffffffff57d24690] (a com.ibm.ws.webcontainer.httpsession.MemorySessionData) at com.ibm.ws.webcontainer.httpsession.SessionData.getValue(SessionData.java:1243) at com.ibm.ws.webcontainer.httpsession.SessionData.getAttribute(SessionData.java:1129) at org.zkoss.zk.ui.http.SimpleSession.getAttribute(SimpleSession.java:218) at org.zkoss.zk.ui.http.SimpleSession.hasAttribute(SimpleSession.java:226) at org.zkoss.zk.ui.http.SimpleSession.hasAttribute(SimpleSession.java:256) at org.zkoss.zk.ui.impl.DesktopImpl.hasAttribute(DesktopImpl.java:587) at org.zkoss.zk.ui.impl.PageImpl.hasAttributeOrFellow(PageImpl.java:389) at org.zkoss.zk.ui.AbstractComponent.hasAttributeOrFellow(AbstractComponent.java:918) at org.zkoss.zk.ui.Components$Wire.containsVariable(Components.java:1068) at org.zkoss.zk.ui.Components$Wire.wireOthers(Components.java:1034) at org.zkoss.zk.ui.Components$Wire.myWireVariables(Components.java:975) at org.zkoss.zk.ui.Components$Wire.wireVariables(Components.java:971) at org.zkoss.zk.ui.Components$Wire.access$500(Components.java:880) at org.zkoss.zk.ui.Components.wireVariables(Components.java:525) at org.zkoss.zk.ui.util.GenericAutowireComposer$BeforeCreateWireListener.onEvent(GenericAutowireComposer.java:232) at org.zkoss.zk.ui.impl.EventProcessor.process0(EventProcessor.java:177) at org.zkoss.zk.ui.impl.EventProcessor.process(EventProcessor.java:140) at org.zkoss.zk.ui.impl.UiEngineImpl.processEvent(UiEngineImpl.java:1602) at org.zkoss.zk.ui.impl.UiEngineImpl.process(UiEngineImpl.java:1386) at org.zkoss.zk.ui.impl.UiEngineImpl.execNewPage0(UiEngineImpl.java:446) at org.zkoss.zk.ui.impl.UiEngineImpl.execNewPage(UiEngineImpl.java:305) at org.zkoss.zk.ui.http.DHtmlLayoutServlet.process(DHtmlLayoutServlet.java:225) at org.zkoss.zk.ui.http.DHtmlLayoutServlet.doGet(DHtmlLayoutServlet.java:146) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:276) at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:463) at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:281) at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:256) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:989) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:501) at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:464) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:113) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:383) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:263) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:195) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:873) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1469)

Greetings,

delete retag edit

## 9 Replies

samchuang
4084 4

Hi

1.

It mentioned
"By default, ZK processes an event in the same Servlet thread that receives the HTTP request. It is the suggested approach because the performance is better and it is easy to integrate with other framework"

Which means

<system-config>
</system-config>


2. Do you use server push ? It may also cause the issue. If you use server push, how do you use it ?

estuardoumg
15 1

Hi samchuang, thanks for your replay.

1. Actually im using the default configuration of ZK 5, i understood zk 5 has the thread event disabled. And for example when a call a modal window or show a message box the thread is not suspended, for that reasson i had to implement code like described in the link http://books.zkoss.org/wiki/ZK_Developer%27s_Reference/UI_Patterns/Event_Threads/Messagebox.

<zk>
<device-config>
<device-type>ajax</device-type>
<timeout-uri>/index.zul</timeout-uri><!-- An empty URL can cause the browser to reload the same URL -->
</device-config>
<session-config>
<session-timeout>1800</session-timeout>
<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>
</session-config>
<!-- Para las pruebas de rendimiento con selenium
<system-config>
<id-generator-class>com.consystec.selenium.idgenerator.ConsystecIdGenerator</id-generator-class>
</system-config> -->
</zk>

2. I don't use server push

I also noticed (Tivoli Performance - WAS) when the application has a low demand works fine (lower than 50 servlet sessions), when the number of sessions grows, begins the hanging threads immediately. Whith the event thread enable it occurs too but occasionally, so I decided to disable the event thread to minimize the incidence but the opposite happened. May be its because the amount of the application users and sessions, but i do not know what else to do.

The application its very simple, has 4 pages and in the last page redirects to others sites (http://www.youtube.com/watch?v=BEJ9HQT1vt8)

estuardoumg
15 1

Anyone has the same problem with too many users ??

samchuang
4084 4

Hi

is't possible to provide a War file that can reproduce the issue ?

by the way, just for curious, do you have any reason for these ?

<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>


bfontana
51 1

estuardoumg
15 1

Something I saw, when the application has the event thread enable the application works fine for a week,
then starts the hung threads. When i changed to event thread disable, the thread pool of websphere was
full in less than 30 min.

This week i'm going to deploy a new version of this application with ZK 5.0.7 EE with event thread enabled
and see what happends in production enviroment. I do not want to ask to the server administrator any load balancing, because it involves
purchasing a license from IBM and maybe serialize everything in my application.

bfontana, what application server are you using ?? Could be websphere application server or could be ZK framework, i'm

samchuang, the reason for these:

<max-desktops-per-session>10</max-desktops-per-session>
<max-requests-per-session>5</max-requests-per-session>

was just testing, my zk.xml just have the default tags and the disable-event-thread tag.

And in the other hand samchuang, i ask to my superior if we can provide the source code but he doesn't let me.

bfontana
51 1

We are using:

Tomcat 5.5.25
Java 1.6 64 bits Sun's JVM (I don't remember update but I could find it if it helps).
RedHat Linux Enteprise Server
"physical" server is a virtualized VMWare VM with 4 Xeon cores assigned and 6 GB RAM.

So I'm afraid is not a WebSphere/ZK issue.

bfontana
51 1

Oh I forgot. We are using ZK 5.0.4 and ZKSpring 3.0-FL-2010-04-20

estuardoumg
15 1

Hi,

Good news, problem solved. I updated the version of zk to 5.0.7, and in the xml just disabled the event threads, taking into consideration all the changes discribed in this seccion and subsection http://books.zkoss.org/wiki/ZK%20Developer's%20Reference/UI%20Patterns/Event%20Threads

Also increase the maximum threads in webcontainer to improve the performance as seen in the Tivoli Performance Viewer.

[hide preview]