-
FEATURED COMPONENTS
First time here? Check out the FAQ!
Hello, we have a situation where Tomcat 6.0.44 sessions are not expiring when due. There is no background "ping" keeping the sessions alive, the TTL is negative (timeout is 20 minutes, sessions run many hours past this). Using the Tomcat Manager app I can see the sessions are beyond the timeout time and I can manually expire them. Histogram dumps show the sessions leave memory, so they are not caught up in a static reference. The Tomcat thread ContainerBackgroundProcessor
is responsible for cleanup of sessions.
The following excerpt from a thread dump shows interaction with Zk. I did another thread dump 50 minutes later and found the same condition. It doesn't directly look like a dead lock but it sure seems to be stuck.
I'm using Zk 3.6.4 (really).
Please suggest a cause for this, it seems the only way to keep the app running is continuously manually expire sessions or restart Tomcat and wait for it to happen again (which took a 8 days).
Thanks!
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fbb320e1800 nid=0x35be in Object.wait() [0x00007fba5ed07000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.zkoss.zk.ui.impl.UiEngineImpl.doActivate(UiEngineImpl.java:1533)
- locked <0x00000005d06078f0> (a java.lang.Object)
at org.zkoss.zk.ui.impl.UiEngineImpl.activate(UiEngineImpl.java:1495)
at org.zkoss.zk.ui.impl.UiEngineImpl.desktopDestroyed(UiEngineImpl.java:180)
at org.zkoss.zk.ui.impl.SimpleDesktopCache.desktopDestroyed(SimpleDesktopCache.java:118)
at org.zkoss.zk.ui.impl.SimpleDesktopCache.stop(SimpleDesktopCache.java:166)
- locked <0x00000005d025c928> (a org.zkoss.zk.ui.impl.SimpleDesktopCache$Cache)
at org.zkoss.zk.ui.impl.SessionDesktopCacheProvider.sessionDestroyed(SessionDesktopCacheProvider.java:59)
at org.zkoss.zk.ui.impl.AbstractWebApp.sessionDestroyed(AbstractWebApp.java:286)
at org.zkoss.zk.ui.http.WebManager.sessionDestroyed(WebManager.java:353)
at org.zkoss.zk.ui.http.HttpSessionListener.sessionDestroyed(HttpSessionListener.java:41)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:721)
- locked <0x00000005d0250e50> (a org.apache.catalina.session.StandardSession)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:588)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:737)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:722)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1371)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645)
at java.lang.Thread.run(Thread.java:701)
With the given information it's hard to give a definite answer but I'll try anyway:
If one thread is blocked while waiting for a resource, there is usually another thread having a lock on the same resource. E.g. in ZK 3.6 is was quite common to use the event processing threads feature (now deprecated).
This feature enabled synchronous blocking Messageboxes, which will block the a server side thread while waiting for user input. To verify you can check if another thread is currently waiting in this line
Then (when the desktop is about to be cleaned up) it will try to gain a lock and wait forever resulting the the stack trace you provided (I am still just making up a case here - this has been made configurable somewhere in ZK 5). It will then wait here (few a method calls deeper according to your stack trace):
https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/SimpleDesktopCache.java#L118 -> https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/UiEngineImpl.java#L180 -> to activate a desktop it will try to get a lock.
However if that's the case you can implement a DesktopCleanup-Listener which is called before trying to activate https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/SimpleDesktopCache.java#L114
In this listener you can try calling this method to unblock any event processing threads on the current desktop still waiting for user input:
org.zkoss.zk.ui.sys.DesktopCtrl.ceaseSuspendedThread(EventProcessingThread, String)
I hope these few ideas give some insight on what might be happening and maybe I was lucky to guess right cause of your locking issue. Otherwise I hope you can provide some additional details on your findings.
Then (when the desktop is about to be cleaned up) it will try to gain a lock and wait forever resulting the the stack trace you provided (I am still just making up a case here - this has been made configurable somewhere in ZK 5). It will then wait here (few a method calls deeper according to your stack trace):
Looking at UiEngineImpl
we see this in doActivate()
:
for (;;) {
final Visualizer old = desktopCtrl.getVisualizer();
if (old == null) break; //grantable
try {
uvlock.wait(120*1000);
} catch (InterruptedException ex) {
throw UiException.Aide.wrap(ex);
}
}
When uvlock
is blocked, this code loops forever as you said. Looking at the traceback, this produces "TIMED_WAITING" rather than "BLOCKED" so it was a bit deceptive. Using a heap dump and the rest of the thread dump I was able to connect the block to a logger that was clogged up.
Thanks for the help.
/Daryl
Asked: 2016-01-21 19:16:37 +0800
Seen: 30 times
Last updated: Jan 27 '16
ZK Clustering, Session Replication Question [closed]
ZK Tomcat 7 Cluster/Session Replication
how to deploy zk application with tomcat startup
Call to zkcomet failing the first time
ZK 6.5.2 and memory leak stopping Tomcat 7.0.32
ZK on tomcat7 not working when accessing by IP
how do I map my domain name to an ZK application in TOMCAT server ...