Overview
The CachedConnectionValve is a JBoss feature that automatically closes all JCA connections whenever a web request completes. Presumably this exists to help novice Java Enterprise programmers (a.k.a. lazy JSP scriplet knuckleheads) remember to close their JDBC connections. Unfortunately this also has the side effect of closing other JCA resources such as JMS connections.
A Troubleshooting Example
Here is an example of the kind of trouble this can cause:
The application in question needs a kind of 'real time' update screen. It's a kind of whiteboard, where multiple users will see the same information updated ever second. I decided to implement this using a JMS topic to broadcast the changes. Each browser would then subscribe to the topic using an AJAX call, then make subsequent AJAX calls to poll the JMS subscription for messages.
The design elements are:
- The server side is an EJB3 SFSB
- The browser invokes the SFSB using Seam Remoting.
- The SFSB is in the Seam conversation, which is in the HttpSession.
So, the problem with this approach was that when the system got to step (3), the JMS connection in the conversation was always closed and the call to receive messages would fail. After looking at the logs and setting some breakpoints I found that CachedConnectionValve was iterating through the list of JCA connections and closing them all during step (2). This of course included the JMS topic connection opened by the SFSB method.
Grepping through the configuration files lead me to this comment in <tt>jbossweb-tomcat55.sar/server.xml}}:
Aha!
Disabling the CachedConnection Manager
- Find the
Valveelement for the CachedConnectionManager injbossweb-tomcat55.sar/server.xmland comment it out: - Locate the Tomcat5 mbean dependency on CachedConnectionManager in
jbossweb-tomcat55.sar/META-INF/jboss-service.xmland comment it out:
5 Comments
Hide/Show CommentsApr 19, 2011
Daniel Holman
As part of a debugging effort, I disabled the CachedConnectionValve. This successfully resolved the bug, but had an unfortunate side effect that produced a new, more serious bug. Somehow, the custom blob handling code in our application had a subtle (and still not understood) dependency on the CachedConnectionValve.
With the CachedConnectionValve disabled, the blob handling code would throw ORA-01002 or ORA-22990 errors depending on the context in which it was called. These errors indicate that the application was attempting to perform database activity across multiple transactions (the activity: blob writing needed to be performed in the same transaction as fetching the blob locator). This indicates that the CachedConnectionValve has some kind of effect on transaction boundaries.
Apr 20, 2011
Joshua Davis
In the example I am assuming that there is at least one transaction per AJAX request, and that the code needs to retain the JMS connection across AJAX calls. CCV will close the JMS connection on every HTTP request, so there is nothing sitting on the server side listening for new messages between AJAX calls.
For JDBC (non-JMS) applications, the CCV is more appropriate, as it is usually a bad idea to keep JDBC connections open across HTTP requests. I haven't looked at the code recently, but I bet it closes any JTA transaction that's open too. Therefore, it sounds to me like you might have a transaction management issue. How does your application manage transactions? If you are required to find the locator and write the blob in the same transaction, and those two operations happen in different HTTP requests, then you will probably need to re-design that.
Apr 20, 2011
Daniel Holman
The problem is almost certainly manifesting itself (in part) as yet another sympyom of long-standing transaction management issues in the application. I'm researching these issues as a side project; our application's transaction management is brittle, but it is reliable enough for a production application (as long as you don't do anything fancy with it). Right now, I'm investigating the CCV because it's a source of some interesting data about the transaction weirdness that's easily accessible.
What's strange about the differences in behavior I'm seeing with the CCV enabled/disabled is that when I investigated what the CCV was doing (reading the source & even stepping through it in a debugger) I could see no reason why the CCV should affect the behavior of the blob writing code; some quick pseudocode for the blob write would look like:
* Get connection from persistence manager
* Use connection to create a prepared statement to SELECT the blob locator
* Fetch the blob locator
* Use connection to create a prepared statement to SELECT the blob locator for UPDATE
What's strange is that it's the last step that I document here that throws the strange ORA codes when the CCV is disabled.
My current guess is that somehow, the reference CCV stores to the JTA transaction (or another unsharable resource) somehow keeps it from being closed, though I'm not yet sure what would be closing the transactions.
As the example clarifies, finding the blob locator and writing the blob do happen in response to the same HTTP request... so while I'm aware of some redesign that's required to our transaction management in the application, I'm currently researching what the extent of that will need to be (and fortunately, it won't involve me debugging AJAX and transaction boundary issues at the same time, which sounds like a pretty grueling task.)
Apr 20, 2011
Joshua Davis
I doubt CCV is hanging on to JTA transactions or anything like that. More likely it was committing then closing the transaction for you. When it does, it complains... tells you to clean up your code.
If all the steps are happening in the same HTTP request, then my example in the wiki page is not very relevant. You're basically having the opposite problem, where CCV was cleaning up the tx/connection for you and now that it isn't doing that you have a problem.
So basically, you have two prepared statements from the same connection that should work, but don't. Are you sure you're not using 'autocommit'? That might cause this kind of problem.
Apr 25, 2011
Daniel Holman
The CCV causes the CachedConnectionManager to hang on to anything that it considers an 'unsharable resource'. I'm sure this could consist of all sorts of resources, but in the application I'm looking at, the only thing that that CCM hangs on to is a reference to the CCV wrapped in an object that's used to modify the behavior of .equals. I don't see any references to either the transaction or the connection itself, so I'm a bit mystified about why disabling the CCV causes the problems I am seeing.
Autocommit isn't on.
Generally speaking, the CCV has more functionality than just closing up JCA connections at the end of web requests, and I wanted to call that out to anyone reading this wiki page-- I was able to resolve problems similar to your AJAX/JMS interaction by turning off the CCV, but this introduced other problems elsewhere. I didn't intend my comment as "Hey, the CCV doesn't do what you say!" because this post did help me resolve a problem, but rather as "The CCV does what you say, but there's more that anyone playing with the CCV should be aware of".