PubMan Indexing Problems

From MPDLMediaWiki
Jump to: navigation, search

Beschreibung

item_container_admin wird mit einer write.lock-Datei permanent geblockt

Fehlerhistorie

  • 16.01.13
    • 14:59 Uhr
    • 15:57 Uhr
  • 17.01.13
    • 11:11 Uhr
  • 18.01.13
    • 12:23 Uhr
  • 22.01.13
    • 15:18 Uhr
    • 16:14 Uhr
  • 23.01.13
    • 13:19 Uhr
    • 17:06 Uhr
  • 24.01.13
    • 17:06 Uhr
  • 28.01.13
    • 15:46 Uhr
  • 29.02.13
    • 11:25 Uhr
  • 30.01.13
    • 11:35 Uhr
  • 31.01.13
    • 13:24 Uhr
    • 17:33 Uhr
  • 01.02.13
    • 08:45 Uhr
    • 09:39 Uhr
  • 04.02.2013
    • 11:10 Uhr
  • 06.02.2013
    • 09:33 Uhr
    • 10:56 Uhr
    • 11:48 Uhr
    • 13:48 Uhr
    • 14:25 Uhr
    • 22:58 Uhr

Infos

  • Scheint eher reproduzierbar mit sehr einfachen Items
  • Sicherheits-Alarm kommt mit 5 Min Verzögerung
  • Timestamp wird für jedes File geändert(evtl. Alarm auf Timestampänderungen eingrenzen?)
  • Framework 1.3.8 verwendet Lucene 3.4.0

Mögliche Fehlergründe

laut:

http://www.gossamer-threads.com/lists/lucene/java-dev/98176

könnte ein NFSNetwork File System (Network File System) stale

A filehandle becomes stale whenever the file or directory referenced by the handle is removed by another host, while your client still holds an active reference to the object. A typical example occurs when the current directory of a process, running on your client, is removed on the server (either by a process running on the server or on another client).

das Problem sein. Mehr zu NFSNetwork File System stales unter:

http://www.cyberciti.biz/tips/nfs-stale-file-handle-error-and-solution.html
https://issues.apache.org/jira/browse/LUCENE-1011

Aus org.apache.lucene.index.IndexFileDeleter :

if (currentCommitPoint == null) {
    // We did not in fact see the segments_N file
    // corresponding to the segmentInfos that was passed
    // in.  Yet, it must exist, because our caller holds
    // the write lock.  This can happen when the directory
    // listing was stale (eg when index accessed via NFSNetwork File System
    // client with stale directory listing cache).  So we
    // try now to explicitly open this commit point:
    SegmentInfos sis = new SegmentInfos();
    try {
        sis.read(directory, segmentInfos.getCurrentSegmentFileName(), codecs);
    } catch (IOException e) {
        throw new CorruptIndexException("failed to locate current segments_N file");
    }
    if (infoStream != null)
    message("forced open of current segments file " + segmentInfos.getCurrentSegmentFileName());
    currentCommitPoint = new CommitPoint(commitsToDelete, directory, sis);
    commits.add(currentCommitPoint);
    incRef(sis, true);
}

Problem: wir verwenden kein NFSNetwork File System. Da dies aber die einzige Stelle in Lucene zu sein scheint, die diese Exception wirft, muss der Fehler auf das System wie ein NFSNetwork File System stale wirkt.

Logfiles

fedoragsearch.log

  • Fehler von LIVE (wurde so auch auf QAQuality Assurance reproduziert bei zwei gleichzeitigen Importen - auf LIVE war kein Import aktiv wenn der Fehler aufgetreten ist)
ERROR 2013-01-17 11:11:37,305 (RESTImpl) dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexReader close error indexName=item_container_admin :; nested exception is: org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file
  • Reproduziert durch batch-delete auf QAQuality Assurance und batch-import auf DEV
2013-01-17 13:29:32,371 (RESTImpl) dk.defxws.fedoragsearch.server.errors.GenericSearchException: updateIndex deletePid error indexName=item_container_admin pid=escidoc:1594805:LV; nested exception is: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/coreservice/server/default/data/index/lucene/item_container_admin/write.lock

indexer_error.log

  • Fehler von LIVE
ERROR 2013-01-14 17:33:14,334 Error on indexing resource escidoc:1481521, index null, all indexes true, action update:
<resultPage>
    <error>
        <message>
        IndexReader close error indexName=item_container_admin :; nested exception is:org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file
        </message>
    </error>
</resultPage>

server.log

  • keine passende Meldung, erst die Folgeerscheinungen sind sichtbar

import-log

  • Warning im Import-Bereich:
TechnicalException: <exception>
<title><h1>500 Internal Application Server Error</h1></title>
<message><p>HTTPHyperText Transfer Protocol connection to &quot;http://localhost:8080/fedoragsearch/rest?operation=updateIndex&amp;action=deletePid&amp;repositoryName=escidocrepository&amp;indexName=content_model_admin&amp;value=escidoc:1594776:LV&quot; failed: &lt;html&gt;&lt;head&gt;&lt;title&gt;JBossWeb/2.0.1.GA - Error report&lt;/title&gt;&lt;style&gt;&lt;!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--&gt;&lt;/style&gt; &lt;/head&gt;&lt;body&gt;&lt;h1&gt;HTTPHyperText Transfer Protocol Status 500 - &lt;/h1&gt;&lt;HR size=&quot;1&quot; noshade=&quot;noshade&quot;&gt;&lt;p&gt;&lt;b&gt;type&lt;/b&gt; Exception report&lt;/p&gt;&lt;p&gt;&lt;b&gt;message&lt;/b&gt; &lt;u&gt;&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;description&lt;/b&gt; &lt;u&gt;The server encountered an internal error () that prevented it from fulfilling this request.&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;exception&lt;/b&gt; &lt;pre&gt;dk.defxws.fedoragsearch.server.errors.ConfigException: get stylesheet from url search/config/rest/:
; nested exception is: 
  dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
  dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:97)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
  dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
&lt;/pre&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;root cause&lt;/b&gt; &lt;pre&gt;dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
  dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:69)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
  dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
&lt;/pre&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;note&lt;/b&gt; &lt;u&gt;The full stack trace of the root cause is available in the JBossWeb/2.0.1.GA logs.&lt;/u&gt;&lt;/p&gt;&lt;HR size=&quot;1&quot; noshade=&quot;noshade&quot;&gt;&lt;h3&gt;JBossWeb/2.0.1.GA&lt;/h3&gt;&lt;/body&gt;&lt;/html&gt;
</p></message>
<class><p>de.escidoc.core.common.exceptions.system.ApplicationServerSystemException</p></class>
<stack-trace><p><![CDATA[
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody2(GsearchHandler.java:205)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody3$advice(GsearchHandler.java:57)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion(GsearchHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:344)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.deleteResource(IndexingHandler.java:243)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody2(IndexingHandler.java:174)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody3$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted(IndexingHandler.java:1)
de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemDeleted(ItemResourceListener.java:171)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete_aroundBody10(FedoraItemHandler.java:479)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure11.run(FedoraItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete(FedoraItemHandler.java:474)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$EnhancerByCGLIB$$f3ee31b5.delete(<generated>)
de.escidoc.core.om.service.ItemHandler.delete_aroundBody4(ItemHandler.java:123)
de.escidoc.core.om.service.ItemHandler$AjcClosure5.run(ItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.service.ItemHandler.delete(ItemHandler.java:119)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody32(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure33.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.aa.security.aop.SecurityInterceptor.proceed(SecurityInterceptor.java:237)
de.escidoc.core.aa.security.aop.SecurityInterceptor.ajc$inlineAccessMethod$de_escidoc_core_aa_security_aop_SecurityInterceptor$de_escidoc_core_aa_security_aop_SecurityInterceptor$proceed(SecurityInterceptor.java:1)
de.escidoc.core.aa.security.aop.SecurityInterceptor.authorize(SecurityInterceptor.java:206)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody34(ItemHandlerBean.java:1)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure35.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.StatisticInterceptor.proceed(StatisticInterceptor.java:205)
de.escidoc.core.common.util.aop.StatisticInterceptor.ajc$inlineAccessMethod$de_escidoc_core_common_util_aop_StatisticInterceptor$de_escidoc_core_common_util_aop_StatisticInterceptor$proceed(StatisticInterceptor.java:1)
de.escidoc.core.common.util.aop.StatisticInterceptor.createStatisticRecord(StatisticInterceptor.java:145)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody36(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure37.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.ejb.ItemHandlerBean.delete(ItemHandlerBean.java:144)
sun.reflect.GeneratedMethodAccessor614.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.invocation.Invocation.performCall(Invocation.java:359)
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
org.jboss.ejb.Container.invoke(Container.java:960)
sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
$Proxy160.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.remoting.rmi.RmiClientInterceptorUtils.invokeRemoteMethod(RmiClientInterceptorUtils.java:111)
org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:99)
org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.invokeInContext(AbstractRemoteSlsbInvokerInterceptor.java:141)
org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.invoke(AbstractSlsbInvokerInterceptor.java:189)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody2(RemoteStatelessEjbProxyFactoryBean.java:171)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody3$advice(RemoteStatelessEjbProxyFactoryBean.java:57)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke(RemoteStatelessEjbProxyFactoryBean.java:1)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
$Proxy253.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor612.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
java.lang.Thread.run(Thread.java:662)
]]></p></stack-trace>
</exception>
  at de.mpg.escidoc.services.pubman.exceptions.ExceptionHandler.handleException(ExceptionHandler.java:93)
AxisFault: <exception>
<title><h1>500 Internal Application Server Error</h1></title>
<message><p>HTTPHyperText Transfer Protocol connection to &quot;http://localhost:8080/fedoragsearch/rest?operation=updateIndex&amp;action=deletePid&amp;repositoryName=escidocrepository&amp;indexName=content_model_admin&amp;value=escidoc:1594776:LV&quot; failed: &lt;html&gt;&lt;head&gt;&lt;title&gt;JBossWeb/2.0.1.GA - Error report&lt;/title&gt;&lt;style&gt;&lt;!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--&gt;&lt;/style&gt; &lt;/head&gt;&lt;body&gt;&lt;h1&gt;HTTPHyperText Transfer Protocol Status 500 - &lt;/h1&gt;&lt;HR size=&quot;1&quot; noshade=&quot;noshade&quot;&gt;&lt;p&gt;&lt;b&gt;type&lt;/b&gt; Exception report&lt;/p&gt;&lt;p&gt;&lt;b&gt;message&lt;/b&gt; &lt;u&gt;&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;description&lt;/b&gt; &lt;u&gt;The server encountered an internal error () that prevented it from fulfilling this request.&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;exception&lt;/b&gt; &lt;pre&gt;dk.defxws.fedoragsearch.server.errors.ConfigException: get stylesheet from url search/config/rest/:
; nested exception is: 
  dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
  dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:97)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
  dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
&lt;/pre&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;root cause&lt;/b&gt; &lt;pre&gt;dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
  dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:69)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
  dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
  dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
  javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
&lt;/pre&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;note&lt;/b&gt; &lt;u&gt;The full stack trace of the root cause is available in the JBossWeb/2.0.1.GA logs.&lt;/u&gt;&lt;/p&gt;&lt;HR size=&quot;1&quot; noshade=&quot;noshade&quot;&gt;&lt;h3&gt;JBossWeb/2.0.1.GA&lt;/h3&gt;&lt;/body&gt;&lt;/html&gt;
</p></message>
<class><p>de.escidoc.core.common.exceptions.system.ApplicationServerSystemException</p></class>
<stack-trace><p><![CDATA[
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody2(GsearchHandler.java:205)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody3$advice(GsearchHandler.java:57)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion(GsearchHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:344)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.deleteResource(IndexingHandler.java:243)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody2(IndexingHandler.java:174)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody3$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted(IndexingHandler.java:1)
de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemDeleted(ItemResourceListener.java:171)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete_aroundBody10(FedoraItemHandler.java:479)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure11.run(FedoraItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete(FedoraItemHandler.java:474)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$EnhancerByCGLIB$$f3ee31b5.delete(<generated>)
de.escidoc.core.om.service.ItemHandler.delete_aroundBody4(ItemHandler.java:123)
de.escidoc.core.om.service.ItemHandler$AjcClosure5.run(ItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.service.ItemHandler.delete(ItemHandler.java:119)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody32(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure33.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.aa.security.aop.SecurityInterceptor.proceed(SecurityInterceptor.java:237)
de.escidoc.core.aa.security.aop.SecurityInterceptor.ajc$inlineAccessMethod$de_escidoc_core_aa_security_aop_SecurityInterceptor$de_escidoc_core_aa_security_aop_SecurityInterceptor$proceed(SecurityInterceptor.java:1)
de.escidoc.core.aa.security.aop.SecurityInterceptor.authorize(SecurityInterceptor.java:206)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody34(ItemHandlerBean.java:1)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure35.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.StatisticInterceptor.proceed(StatisticInterceptor.java:205)
de.escidoc.core.common.util.aop.StatisticInterceptor.ajc$inlineAccessMethod$de_escidoc_core_common_util_aop_StatisticInterceptor$de_escidoc_core_common_util_aop_StatisticInterceptor$proceed(StatisticInterceptor.java:1)
de.escidoc.core.common.util.aop.StatisticInterceptor.createStatisticRecord(StatisticInterceptor.java:145)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody36(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure37.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.ejb.ItemHandlerBean.delete(ItemHandlerBean.java:144)
sun.reflect.GeneratedMethodAccessor614.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.invocation.Invocation.performCall(Invocation.java:359)
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
org.jboss.ejb.Container.invoke(Container.java:960)
sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
$Proxy160.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.remoting.rmi.RmiClientInterceptorUtils.invokeRemoteMethod(RmiClientInterceptorUtils.java:111)
org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:99)
org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.invokeInContext(AbstractRemoteSlsbInvokerInterceptor.java:141)
org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.invoke(AbstractSlsbInvokerInterceptor.java:189)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody2(RemoteStatelessEjbProxyFactoryBean.java:171)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody3$advice(RemoteStatelessEjbProxyFactoryBean.java:57)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke(RemoteStatelessEjbProxyFactoryBean.java:1)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
$Proxy253.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor612.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
java.lang.Thread.run(Thread.java:662)
]]></p></stack-trace>
</exception>
  at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)

Experimente

  • fgsindex.defaultWriteLockTimeout = 3000 (msec anstelle von 100.000.000) wait time for an IndexWriter for the same directory in case of a write.lock before throwing a LockObtainFailedException
de.mpg.escidoc.services.common.exceptions.TechnicalException: <exception>
  <title><h1>500 Internal eSciDocEnhanced Scientific Documentation System Error</h1></title>
  <message><p>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTFUnicode Transformation Format-8&quot;?&gt;
&lt;resultPage&gt;
&lt;error&gt;
&lt;message&gt;Update error pidOrFilename=escidoc:1602273; nested exception is:
        dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexWriter new error, creating index indexName=item_container_admin :
; nested exception is:
        org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/coreservice/server/default/data/index/lucene/item_container_admin/write.lock&lt;/message&gt;
&lt;/error&gt;
&lt;/resultPage&gt;
</p></message>
  <class><p>de.escidoc.core.common.exceptions.system.SystemException</p></class>
  <stack-trace><p><![CDATA[
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody8(IndexingHandler.java:502)
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody9$advice(IndexingHandler.java:57)
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:338)
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
    de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
    de.escidoc.core.common.business.indexing.IndexingHandler.indexResource(IndexingHandler.java:285)
    de.escidoc.core.common.business.indexing.IndexingHandler.addResource(IndexingHandler.java:228)
    de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated_aroundBody0(IndexingHandler.java:153)
    de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated_aroundBody1$advice(IndexingHandler.java:57)
    de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated(IndexingHandler.java:1)
    de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemCreated(ItemResourceListener.java:98)
    de.escidoc.core.om.business.fedora.item.FedoraItemHandler.create_aroundBody6(FedoraItemHandler.java:434)
    de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure7.run(FedoraItemHandler.java:1)
    org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
    de.escidoc.core.om.business.fedora.item.FedoraItemHandler.create(FedoraItemHandler.java:409)
    de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
    net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
    org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
    de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
    org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
    de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
  • project FgsLucene OperationsImpl.java ohne closeIndexReader
ERROR 2013-02-12 08:35:14,415 [http-0.0.0.0-8080-8] (RESTImpl) RESTImpl error:
dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexReader open error indexName=item_container_admin :
; nested exception is:
        java.io.FileNotFoundException: /data/coreservice/server/default/data/index/lucene/item_container_admin/_7690.prx (Too many open files)
        at dk.defxws.fgslucene.OperationsImpl.getIndexReader(OperationsImpl.java:679)
        at dk.defxws.fgslucene.OperationsImpl.updateIndex(OperationsImpl.java:265)
        at dk.defxws.fedoragsearch.server.GenericOperationsImpl.updateIndex(GenericOperationsImpl.java:356)
        at dk.defxws.fedoragsearch.server.RESTImpl.updateIndex(RESTImpl.java:280)
        at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:117)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:662)