Anyway, when you hit a problem keep in mind that you're never the only person to have encountered this problem. Somebody else has seen it and probably been in a more desperate situation to need a solution.
Every once in a while I'd see some odd stuff going on where a bunch of threads seemed to get stuck. Those threads interacted via a CORBA interface, so I just chalked it up to some CORBA-wackiness and thought I'd try to code a way to kill off those threads when they got stuck waiting for CORBA to finish.
Sidenote: CORBA came from the OMG, which has the most ironically appropriate acronym ever invented, because "it's like, OMG, could there be anything better than a consortium named OMG!?" By the way, do not click over to the talk page on Wikipedia entry for OMG if you're in a quiet professional environment -- I started laughing so much my wife thought there was something wrong with me.
So anyway, I got a recent thread dump (not a stacktrace, since those threads didn't throw any exceptions, they were just waiting) and decided to look around a bit for the answer. Here's a sample of one of the stuck CORBA threads:
I stumbled around the Interwebs for a bit and happened upon the inevitable similar stack traces and mentions in a forum or two or three, but none of them had the answer. Then I expanded my search a bit and found an existing item (7016182) in the Java bug database... submitted about a year ago and a status of "Cause Known" but now what? So I searched on the related bug ID numbers and found yet more info in number 7046238. Yay! Finally an answer! It says it's been fixed in JDK release 6u27 which I'm pretty sure is 1 update newer than what we were running.
That's good news, but it still needs to get tested. In the meantime, I compared the source of the affected JDK class CorbaClientRequestDispatcherImpl with the the fixed version. Yep, just as the bugfix note said, the fix is a change to how the synchronization is handled and it makes me glad I didn't have to try to figure out how to fix it. In a way it's comforting to know that even the experts sometimes make mistakes.
Anyway, that was my adventure this weekend. What did you do?
Sidenote: CORBA came from the OMG, which has the most ironically appropriate acronym ever invented, because "it's like, OMG, could there be anything better than a consortium named OMG!?" By the way, do not click over to the talk page on Wikipedia entry for OMG if you're in a quiet professional environment -- I started laughing so much my wife thought there was something wrong with me.
So anyway, I got a recent thread dump (not a stacktrace, since those threads didn't throw any exceptions, they were just waiting) and decided to look around a bit for the answer. Here's a sample of one of the stuck CORBA threads:
"pool-538-thread-1" - Thread t@67441
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <5dfd006a> (a java.lang.Object)
at java.lang.Object.wait(Object. java:485)
at com.sun.corba.se.impl. transport. CorbaResponseWaitingRoomImpl. waitForResponse( CorbaResponseWaitingRoomImpl. java:140)
at com.sun.corba.se.impl. transport. SocketOrChannelConnectionImpl. waitForResponse( SocketOrChannelConnectionImpl. java:1084)
at com.sun.corba.se.impl. protocol. CorbaMessageMediatorImpl. waitForResponse( CorbaMessageMediatorImpl.java: 253)
at com.sun.corba.se.impl. protocol. CorbaClientRequestDispatcherIm pl.marshalingComplete1( CorbaClientRequestDispatcherIm pl.java:362)
at com.sun.corba.se.impl. protocol. CorbaClientRequestDispatcherIm pl.marshalingComplete( CorbaClientRequestDispatcherIm pl.java:336)
at com.sun.corba.se.impl. protocol. CorbaClientDelegateImpl. invoke( CorbaClientDelegateImpl.java: 129)
at com.sun.corba.se.impl.corba. RequestImpl.doInvocation( RequestImpl.java:309)
at com.sun.corba.se.impl.corba. RequestImpl.invoke( RequestImpl.java:230)
- locked <6dae8354> (a com.sun.corba.se.impl.corba. RequestImpl)
[etc., etc...]
I stumbled around the Interwebs for a bit and happened upon the inevitable similar stack traces and mentions in a forum or two or three, but none of them had the answer. Then I expanded my search a bit and found an existing item (7016182) in the Java bug database... submitted about a year ago and a status of "Cause Known" but now what? So I searched on the related bug ID numbers and found yet more info in number 7046238. Yay! Finally an answer! It says it's been fixed in JDK release 6u27 which I'm pretty sure is 1 update newer than what we were running.
That's good news, but it still needs to get tested. In the meantime, I compared the source of the affected JDK class CorbaClientRequestDispatcherImpl with the the fixed version. Yep, just as the bugfix note said, the fix is a change to how the synchronization is handled and it makes me glad I didn't have to try to figure out how to fix it. In a way it's comforting to know that even the experts sometimes make mistakes.
Anyway, that was my adventure this weekend. What did you do?


