Question : Application Deadlock Exception in JBOSS

Hi experts,
Iam facing an issue with jboss and ejbs. Iam getting the TransactionRolledBackLocalException with one of my clients. The wierd thing is this happens not very often:  Please find the stack trace below:


TransactionRolledbackLocalException in method: public abstract java.lang.Integer com.cmt.tower.report.interfaces.ReportRecordLocal.getId(), causedBy:
org.jboss.util.deadlock.ApplicationDeadlockException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1441d9b, bean=ReportRecord, id=10221, refs=2, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=production/173677, BranchQual=, localId=173677], synched=Thread[http-0.0.0.0-8080-9,5,jboss], timeout=5000, queue=[], holder=TransactionImpl:XidImpl[FormatId=257, GlobalId=production/173675, BranchQual=, localId=173675], waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@f3add0, bean=PersonRecord, id=19750, refs=2, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=production/173675, BranchQual=, localId=173675], synched=null, timeout=5000, queue=[TXLOCK waitingTx=TransactionImpl:XidImpl[FormatId=257, GlobalId=production/173677, BranchQual=, localId=173677] id=0 thread=Thread[http-0.0.0.0-8080-6,5,jboss] queued=true], waitingResourceHolder=TransactionImpl:XidImpl[FormatId=257, GlobalId=production/173675, BranchQual=, localId=173675]
      at org.jboss.util.deadlock.DeadlockDetector.deadlockDetection(DeadlockDetector.java:69)
      at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.waitForTx(QueuedPessimisticEJBLock.java:292)
      at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:230)
      at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java:194)
      at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:100)
      at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:68)
      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:136)
      at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:527)
      at org.jboss.ejb.Container.invoke(Container.java:954)
      at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
      at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java:65)
      at $Proxy167.getId(Unknown Source)
      at com.cmt.tower.report.ejb.ReportManagerBean.assembleReportObject(ReportManagerBean.java:2651)
      at com.cmt.tower.report.ejb.ReportManagerBean.findReports(ReportManagerBean.java:1304)
      at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:136)
      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
      at org.jboss.ejb.Container.invoke(Container.java:954)
      at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
      at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
      at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
      at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
      at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
      at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
      at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
      at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
      at $Proxy229.findReports(Unknown Source)
      at com.cmt.tower.delegate.client.ClientReportManagerBD.findReports(ClientReportManagerBD.java:372)
      at com.cmt.tower.servlet.officer.ShowOfficerHomeServlet.doPost(ShowOfficerHomeServlet.java:256)
      at com.cmt.tower.servlet.TowerServlet.doGet(TowerServlet.java:297)
      at javax.servlet.http.HttpServlet.doHead(HttpServlet.java:270)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:714)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at com.cmt.tower.servlet.UserFilter.doFilter(UserFilter.java:244)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
      at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
      at java.lang.Thread.run(Thread.java:595)

Any help would be appreciated, I have searched the whole net for this, but couldnt get much information on it.

Thanks in advance.

Code Snippet:

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: 55: 56: 57: 58: 59: 60: 61: 62: 63: 64: 65: 66: 67: 68: 69: 70: 71: 72: 73: 74: 75: 76: 77: 78: 79: 80: 81: 82: 83: 84: 85: 86: 87: 88: 89: 90: 91: 92: 93: 94: 95: 96: 97: 98: 99: 100: 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 126: 127: 128: 129: 130: 131: 132: 133: 134: 135: 136: 137: 138: 139: 140: 141: 142: 143: 144: 145: 146: 147: 148: 149: 150: 151: 152: 153: 154: 155: 156: 157: 158: 159: 160: 161: 162: 163: 164: 165: 166: 167: 168: 169: 170: 171: 172: 173: 174: 175: 176: 177: 178: 179: 180: 181: 182: 183: 184: 185: 186: 187: 188: 189: 190: 191: 192: 193: 194: 195: 196: 197: 198: 199: 200: 201: 202: 203: 204: 205: 206: 207: 208: 209: 210: 211: 212: 213: 214: 215: 216: 217: 218: 219: 220: 221: 222: 223: 224: 225: 226: 227: 228: 229: 230: 231: 232: 233: 234: 235: 236: 237: 238: 239: 240: 241: 242: 243: 244: 245: 246: 247: 248: 249: 250: 251: 252: 253: 254: 255: 256: 257: 258: 259: 260: 261: 262: 263: 264: 265: 266: 267: 268: 269: 270: 271: 272: 273: 274: 275: 276: 277: 278: 279: 280: 281: 282: 283: 284:

This is the code where Iam getting this error mostly: this is the function from "ReportManagerBean.java" public synchronized void updateReport(Report report) throws ReportNotFoundException, ReportUpdateException { ReportRecordLocal reportRec; Log log = LogFactory.getFactory().getInstance(this.getClass()); try { reportRec = reportRecHome.findByPrimaryKey(report.getId()); } catch (FinderException fe) { if (fe instanceof ObjectNotFoundException) { throw new ReportNotFoundException(fe); } else { String msg = "error finding report: " + report.getId(); log.error(msg, fe); throw new EJBException(fe); } } // update the report record reportRec.setBeat(report.getBeat()); reportRec.setCaseNumber(report.getCaseNumber()); reportRec.setChargeType(report.getChargeType()); reportRec.setDetail(report.getDetail()); reportRec.setDisposition(report.getDisposition()); reportRec.setIncidentDate(report.getIncidentDate()); reportRec.setLocation(report.getLocation()); reportRec.setReportDate(report.getReportDate()); reportRec.setConfidential(report.getConfidential()); reportRec.setIncident(report.getIncident()); reportRec.setPlaceOfOccurrence(report.getPlaceOfOccurrence()); reportRec.setTimeCommitted(report.getTimeCommitted()); reportRec.setDateCommitted(report.getDateCommitted()); reportRec.setOic(report.getOic()); reportRec.setFormType(report.getFormType()); reportRec.setAgeClass(report.getAgeClass()); reportRec.setSupplement(report.isSupplement()); reportRec.setStatus(report.getStatus()); reportRec.setComplainantName(report.getComplainantName()); reportRec.setErrors(report.hasErrors()); reportRec.setDateLocked(report.getDateLocked()); reportRec.setFlags(report.getFlags()); // update the primary document if ( report.getPrimaryDocument() != null ) { FileInfo file = report.getPrimaryDocument(); RepositoryFileRecordLocal fileRec = reportRec.getPrimaryDocument(); fileRec.setCreatedBy( file.getCreatedBy() ); fileRec.setDateModified( file.getDateModified() ); fileRec.setMimeType( file.getMimeType() ); fileRec.setName( file.getName() ); } // update the primary document if ( report.getFinalDocument() != null ) { FileInfo file = report.getFinalDocument(); RepositoryFileRecordLocal fileRec = reportRec.getFinalDocument(); fileRec.setCreatedBy( file.getCreatedBy() ); fileRec.setDateModified( file.getDateModified() ); fileRec.setMimeType( file.getMimeType() ); fileRec.setName( file.getName() ); } // create the persons Collection persons = report.getPersons(); //Collection personRecs = reportRec.getPersons(); // BEGIN TEMP Collection personRecs = reportRec.getPersonAssoc(); Iterator personRecsItr = personRecs.iterator(); Collection tempCollection = new HashSet(); while (personRecsItr.hasNext()) { PersonReportRecordLocal pr = (PersonReportRecordLocal) personRecsItr.next(); try { tempCollection.add(personRecHome.findByPrimaryKey(pr.getPersonId())); } catch (Exception ex) { ; } } personRecs = tempCollection; // END TEMP Set recsToRemove = new HashSet(); recsToRemove.addAll(personRecs); if (persons != null) { Iterator iter = persons.iterator(); try{ while (iter.hasNext()) { Person person = (Person) iter.next(); if ( person.getFirstName() == null && person.getMiddleName() == null && person.getLastName() == null ) continue; PersonRecordLocal personRec = null; if ( person.getId() != null ) { // look for an existing person record //Iterator prIter = reportRec.getPersons().iterator(); //BEGIN TEMP Iterator prIter = personRecs.iterator(); //END TEMP while ( prIter.hasNext() ) { PersonRecordLocal pr = (PersonRecordLocal) prIter.next(); if ( pr.getId() == person.getId() ) { personRec = pr; break; } } } // query database for existing person Map personQuery = new HashMap(); Object[] args = new Object[4]; if (person.getLastName() == null) person.setLastName(""); args[0] = person.getLastName().toLowerCase() + "%"; if (person.getFirstName() == null) person.setFirstName(""); args[1] = person.getFirstName().toLowerCase() + "%"; if (person.getMiddleName() == null) person.setMiddleName(""); args[2] = person.getMiddleName().toLowerCase() + "%"; if (person.getSsn() == null) person.setSsn(""); args[3] = person.getSsn().toLowerCase() + "%"; // check if person already exists in database // by full name and ssn String queryBase = "SELECT OBJECT(p) FROM PersonRecord p WHERE " + "LCASE(p.lastName) LIKE ?1" + "AND " + "LCASE(p.firstName) LIKE ?2" + "AND " + "LCASE(p.middleName) LIKE ?3" + "AND " + "LCASE(p.ssn) LIKE ?4"; Collection existingPersons = personRecHome.findPersons(queryBase, args); if (existingPersons != null) { Iterator prIter = existingPersons.iterator(); if ( prIter.hasNext() ) { PersonRecordLocal pr = (PersonRecordLocal) prIter.next(); person.setId(pr.getId()); personRec = pr; } } else { // check if person already exists in database // only by full name queryBase = "SELECT OBJECT(p) FROM PersonRecord p WHERE " + "LCASE(p.lastName) LIKE ?1" + "AND " + "LCASE(p.firstName) LIKE ?2" + "AND " + "LCASE(p.middleName) LIKE ?3"; Object[] nameArgs = new Object[3]; for (int i=0; i < 3; i++) nameArgs[i] = args[i]; existingPersons = personRecHome.findPersons(queryBase, nameArgs); if (existingPersons != null) { Iterator prIter = existingPersons.iterator(); if ( prIter.hasNext() ) { PersonRecordLocal pr = (PersonRecordLocal) prIter.next(); person.setId(pr.getId()); personRec = pr; } } } if ( personRec == null ) { personRec = personRecHome.create( person.getFirstName(), person.getMiddleName(), person.getLastName(), person.getAddress(), person.getPhone(), person.getDob(), person.getSex(), person.getRace(), person.getHeight(), person.getWeight(), person.getHair(), person.getEyes(), person.getSsn()); // add the person to the person set personRecs.add( personRec ); } else { if (person.getFirstName() != null) personRec.setFirstName( person.getFirstName() ); if (person.getMiddleName() != null) personRec.setMiddleName( person.getMiddleName() ); if (person.getLastName() != null) personRec.setLastName( person.getLastName() ); if (person.getAddress() != null) personRec.setAddress( person.getAddress() ); if (person.getPhone() != null) personRec.setPhone( person.getPhone() ); if (person.getDob() != null) personRec.setDob( person.getDob() ); if (person.getSex() != null) personRec.setSex( person.getSex() ); if (person.getRace() != null) personRec.setRace( person.getRace() ); if (person.getHeight() != null) personRec.setHeight( person.getHeight() ); if (person.getWeight() != null) personRec.setWeight( person.getWeight() ); if (person.getHair() != null) personRec.setHair( person.getHair() ); if (person.getEyes() != null) personRec.setEyes( person.getEyes() ); if (person.getSsn() != null) personRec.setSsn( person.getSsn() ); recsToRemove.remove(personRec); } // create a person report reference for this entry // if one doesn't already exist try { PersonReportRecordLocal existingRef = personReportRecHome.findByPrimaryKey( new PersonReportRecordPK( personRec.getId(), report.getId()) ); } catch (FinderException fe) { PersonReportRecordLocal personReportRec = personReportRecHome.create( personRec.getId(), report.getId()); } Collection roles = person.getRoles(); if (roles != null) { Collection personRecRoles = personRec.getRoles(); Iterator prrIter = personRecRoles.iterator(); Set oldRoles = new HashSet(); while ( prrIter.hasNext() ) { PersonRoleRecordLocal prr = (PersonRoleRecordLocal) prrIter.next(); if ( ! roles.contains( prr.getRole() ) ) { prrIter.remove(); prr.remove(); } else { oldRoles.add( prr.getRole() ); } } Iterator roleIter = roles.iterator(); while (roleIter.hasNext()) { String role = (String) roleIter.next(); if ( ! oldRoles.contains( role )) { PersonRoleRecordLocal roleRec = roleRecHome.create( personRec.getId(), role); personRecRoles.add(roleRec); } } } else { Iterator prrIter = personRec.getRoles().iterator(); while ( prrIter.hasNext() ) { PersonRoleRecordLocal prr = (PersonRoleRecordLocal) prrIter.next(); prrIter.remove(); prr.remove(); } } } } catch (Exception ce) { log.error( ce ); throw new ReportUpdateException( "error updating report " + report.getId() ); } }

Open in New Window Select All

Answer : Application Deadlock Exception in JBOSS

and also you are doing second like query on another table:
SELECT OBJECT(p) FROM PersonRecord p WHERE " +
                                                                  "LCASE(p.lastName) LIKE ?1" + "AND " +
                                                               "LCASE(p.firstName) LIKE ?2" + "AND " +
                                                               "LCASE(p.middleName) LIKE ?3";

So I would suggest to review the way of finding existing customers.
for every person in report you are doing two such a heavy queries.
and if in report you have more persons... just multiple it and you will come with very big wait seonds.