0

Tomcat sessions not expiring

asked 2016-01-21 19:16:37 +0800

dastultz gravatar image dastultz
797 9

Hello, we have a situation where Tomcat 6.0.44 sessions are not expiring when due. There is no background "ping" keeping the sessions alive, the TTL is negative (timeout is 20 minutes, sessions run many hours past this). Using the Tomcat Manager app I can see the sessions are beyond the timeout time and I can manually expire them. Histogram dumps show the sessions leave memory, so they are not caught up in a static reference. The Tomcat thread ContainerBackgroundProcessor is responsible for cleanup of sessions.

The following excerpt from a thread dump shows interaction with Zk. I did another thread dump 50 minutes later and found the same condition. It doesn't directly look like a dead lock but it sure seems to be stuck.

I'm using Zk 3.6.4 (really).

Please suggest a cause for this, it seems the only way to keep the app running is continuously manually expire sessions or restart Tomcat and wait for it to happen again (which took a 8 days).

Thanks!

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fbb320e1800 nid=0x35be in Object.wait() [0x00007fba5ed07000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.zkoss.zk.ui.impl.UiEngineImpl.doActivate(UiEngineImpl.java:1533) - locked <0x00000005d06078f0> (a java.lang.Object) at org.zkoss.zk.ui.impl.UiEngineImpl.activate(UiEngineImpl.java:1495) at org.zkoss.zk.ui.impl.UiEngineImpl.desktopDestroyed(UiEngineImpl.java:180) at org.zkoss.zk.ui.impl.SimpleDesktopCache.desktopDestroyed(SimpleDesktopCache.java:118) at org.zkoss.zk.ui.impl.SimpleDesktopCache.stop(SimpleDesktopCache.java:166) - locked <0x00000005d025c928> (a org.zkoss.zk.ui.impl.SimpleDesktopCache$Cache) at org.zkoss.zk.ui.impl.SessionDesktopCacheProvider.sessionDestroyed(SessionDesktopCacheProvider.java:59) at org.zkoss.zk.ui.impl.AbstractWebApp.sessionDestroyed(AbstractWebApp.java:286) at org.zkoss.zk.ui.http.WebManager.sessionDestroyed(WebManager.java:353) at org.zkoss.zk.ui.http.HttpSessionListener.sessionDestroyed(HttpSessionListener.java:41) at org.apache.catalina.session.StandardSession.expire(StandardSession.java:721) - locked <0x00000005d0250e50> (a org.apache.catalina.session.StandardSession) at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:588) at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:737) at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:722) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1371) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645) at java.lang.Thread.run(Thread.java:701)

delete flag offensive retag edit

2 Answers

Sort by ยป oldest newest most voted
1

answered 2016-01-25 02:19:04 +0800

cor3000 gravatar image cor3000
6280 2 7

updated 2016-01-25 02:20:39 +0800

With the given information it's hard to give a definite answer but I'll try anyway:

If one thread is blocked while waiting for a resource, there is usually another thread having a lock on the same resource. E.g. in ZK 3.6 is was quite common to use the event processing threads feature (now deprecated).

This feature enabled synchronous blocking Messageboxes, which will block the a server side thread while waiting for user input. To verify you can check if another thread is currently waiting in this line

https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/EventProcessingThreadImpl.java#L221

Then (when the desktop is about to be cleaned up) it will try to gain a lock and wait forever resulting the the stack trace you provided (I am still just making up a case here - this has been made configurable somewhere in ZK 5). It will then wait here (few a method calls deeper according to your stack trace):

https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/SimpleDesktopCache.java#L118 -> https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/UiEngineImpl.java#L180 -> to activate a desktop it will try to get a lock.

However if that's the case you can implement a DesktopCleanup-Listener which is called before trying to activate https://github.com/zkoss/zk/blob/v3.6.4/zk/src/org/zkoss/zk/ui/impl/SimpleDesktopCache.java#L114

In this listener you can try calling this method to unblock any event processing threads on the current desktop still waiting for user input:

org.zkoss.zk.ui.sys.DesktopCtrl.ceaseSuspendedThread(EventProcessingThread, String)

I hope these few ideas give some insight on what might be happening and maybe I was lucky to guess right cause of your locking issue. Otherwise I hope you can provide some additional details on your findings.

link publish delete flag offensive edit
0

answered 2016-01-27 14:49:44 +0800

dastultz gravatar image dastultz
797 9

Then (when the desktop is about to be cleaned up) it will try to gain a lock and wait forever resulting the the stack trace you provided (I am still just making up a case here - this has been made configurable somewhere in ZK 5). It will then wait here (few a method calls deeper according to your stack trace):

Looking at UiEngineImpl we see this in doActivate():

for (;;) {
final Visualizer old = desktopCtrl.getVisualizer();
if (old == null) break; //grantable

try {
    uvlock.wait(120*1000);
} catch (InterruptedException ex) {
    throw UiException.Aide.wrap(ex);
}
}

When uvlock is blocked, this code loops forever as you said. Looking at the traceback, this produces "TIMED_WAITING" rather than "BLOCKED" so it was a bit deceptive. Using a heap dump and the rest of the thread dump I was able to connect the block to a logger that was clogged up.

Thanks for the help.

/Daryl

link publish delete flag offensive edit

Comments

you're welcome I am glad my reply wasn't totally misleading ;) and helped you to identify the thread keeping the lock. Robert

cor3000 ( 2016-01-29 02:32:22 +0800 )edit
Your answer
Please start posting your answer anonymously - your answer will be saved within the current session and published after you log in or create a new account. Please try to give a substantial answer, for discussions, please use comments and please do remember to vote (after you log in)!

[hide preview]

Question tools

Follow
1 follower

RSS

Stats

Asked: 2016-01-21 19:16:37 +0800

Seen: 30 times

Last updated: Jan 27 '16

Support Options
  • Email Support
  • Training
  • Consulting
  • Outsourcing
Learn More