PubMan Indexing Problems
Jump to navigation
Jump to search
Beschreibung[edit]
item_container_admin wird mit einer write.lock-Datei permanent geblockt
Fehlerhistorie[edit]
- 16.01.13
- 14:59 Uhr
- 15:57 Uhr
- 17.01.13
- 11:11 Uhr
- 18.01.13
- 12:23 Uhr
- 22.01.13
- 15:18 Uhr
- 16:14 Uhr
- 23.01.13
- 13:19 Uhr
- 17:06 Uhr
- 24.01.13
- 17:06 Uhr
- 28.01.13
- 15:46 Uhr
- 29.02.13
- 11:25 Uhr
- 30.01.13
- 11:35 Uhr
- 31.01.13
- 13:24 Uhr
- 17:33 Uhr
- 01.02.13
- 08:45 Uhr
- 09:39 Uhr
- 04.02.2013
- 11:10 Uhr
- 06.02.2013
- 09:33 Uhr
- 10:56 Uhr
- 11:48 Uhr
- 13:48 Uhr
- 14:25 Uhr
- 22:58 Uhr
Infos[edit]
- Scheint eher reproduzierbar mit sehr einfachen Items
- Sicherheits-Alarm kommt mit 5 Min Verzögerung
- Timestamp wird für jedes File geändert(evtl. Alarm auf Timestampänderungen eingrenzen?)
- Framework 1.3.8 verwendet Lucene 3.4.0
Mögliche Fehlergründe[edit]
laut:
könnte ein NFS (Network File System) stale
- A filehandle becomes stale whenever the file or directory referenced by the handle is removed by another host, while your client still holds an active reference to the object. A typical example occurs when the current directory of a process, running on your client, is removed on the server (either by a process running on the server or on another client).
das Problem sein. Mehr zu NFS stales unter:
- http://www.cyberciti.biz/tips/nfs-stale-file-handle-error-and-solution.html
- https://issues.apache.org/jira/browse/LUCENE-1011
Aus org.apache.lucene.index.IndexFileDeleter :
if (currentCommitPoint == null) {
// We did not in fact see the segments_N file
// corresponding to the segmentInfos that was passed
// in. Yet, it must exist, because our caller holds
// the write lock. This can happen when the directory
// listing was stale (eg when index accessed via NFS
// client with stale directory listing cache). So we
// try now to explicitly open this commit point:
SegmentInfos sis = new SegmentInfos();
try {
sis.read(directory, segmentInfos.getCurrentSegmentFileName(), codecs);
} catch (IOException e) {
throw new CorruptIndexException("failed to locate current segments_N file");
}
if (infoStream != null)
message("forced open of current segments file " + segmentInfos.getCurrentSegmentFileName());
currentCommitPoint = new CommitPoint(commitsToDelete, directory, sis);
commits.add(currentCommitPoint);
incRef(sis, true);
}
Problem: wir verwenden kein NFS. Da dies aber die einzige Stelle in Lucene zu sein scheint, die diese Exception wirft, muss der Fehler auf das System wie ein NFS stale wirkt.
Logfiles[edit]
fedoragsearch.log[edit]
- Fehler von LIVE (wurde so auch auf QA reproduziert bei zwei gleichzeitigen Importen - auf LIVE war kein Import aktiv wenn der Fehler aufgetreten ist)
ERROR 2013-01-17 11:11:37,305 (RESTImpl) dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexReader close error indexName=item_container_admin :; nested exception is: org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file
- Reproduziert durch batch-delete auf QA und batch-import auf DEV
2013-01-17 13:29:32,371 (RESTImpl) dk.defxws.fedoragsearch.server.errors.GenericSearchException: updateIndex deletePid error indexName=item_container_admin pid=escidoc:1594805:LV; nested exception is: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/coreservice/server/default/data/index/lucene/item_container_admin/write.lock
indexer_error.log[edit]
- Fehler von LIVE
ERROR 2013-01-14 17:33:14,334 Error on indexing resource escidoc:1481521, index null, all indexes true, action update:
<resultPage>
<error>
<message>
IndexReader close error indexName=item_container_admin :; nested exception is:org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file
</message>
</error>
</resultPage>
server.log[edit]
- keine passende Meldung, erst die Folgeerscheinungen sind sichtbar
import-log[edit]
- Warning im Import-Bereich:
TechnicalException: <exception>
<title><h1>500 Internal Application Server Error</h1></title>
<message><p>HTTP connection to "http://localhost:8080/fedoragsearch/rest?operation=updateIndex&action=deletePid&repositoryName=escidocrepository&indexName=content_model_admin&value=escidoc:1594776:LV" failed: <html><head><title>JBossWeb/2.0.1.GA - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>dk.defxws.fedoragsearch.server.errors.ConfigException: get stylesheet from url search/config/rest/:
; nested exception is:
dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:97)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
</pre></p><p><b>root cause</b> <pre>dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:69)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the JBossWeb/2.0.1.GA logs.</u></p><HR size="1" noshade="noshade"><h3>JBossWeb/2.0.1.GA</h3></body></html>
</p></message>
<class><p>de.escidoc.core.common.exceptions.system.ApplicationServerSystemException</p></class>
<stack-trace><p><![CDATA[
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody2(GsearchHandler.java:205)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody3$advice(GsearchHandler.java:57)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion(GsearchHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:344)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.deleteResource(IndexingHandler.java:243)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody2(IndexingHandler.java:174)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody3$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted(IndexingHandler.java:1)
de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemDeleted(ItemResourceListener.java:171)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete_aroundBody10(FedoraItemHandler.java:479)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure11.run(FedoraItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete(FedoraItemHandler.java:474)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$EnhancerByCGLIB$$f3ee31b5.delete(<generated>)
de.escidoc.core.om.service.ItemHandler.delete_aroundBody4(ItemHandler.java:123)
de.escidoc.core.om.service.ItemHandler$AjcClosure5.run(ItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.service.ItemHandler.delete(ItemHandler.java:119)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody32(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure33.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.aa.security.aop.SecurityInterceptor.proceed(SecurityInterceptor.java:237)
de.escidoc.core.aa.security.aop.SecurityInterceptor.ajc$inlineAccessMethod$de_escidoc_core_aa_security_aop_SecurityInterceptor$de_escidoc_core_aa_security_aop_SecurityInterceptor$proceed(SecurityInterceptor.java:1)
de.escidoc.core.aa.security.aop.SecurityInterceptor.authorize(SecurityInterceptor.java:206)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody34(ItemHandlerBean.java:1)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure35.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.StatisticInterceptor.proceed(StatisticInterceptor.java:205)
de.escidoc.core.common.util.aop.StatisticInterceptor.ajc$inlineAccessMethod$de_escidoc_core_common_util_aop_StatisticInterceptor$de_escidoc_core_common_util_aop_StatisticInterceptor$proceed(StatisticInterceptor.java:1)
de.escidoc.core.common.util.aop.StatisticInterceptor.createStatisticRecord(StatisticInterceptor.java:145)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody36(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure37.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.ejb.ItemHandlerBean.delete(ItemHandlerBean.java:144)
sun.reflect.GeneratedMethodAccessor614.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.invocation.Invocation.performCall(Invocation.java:359)
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
org.jboss.ejb.Container.invoke(Container.java:960)
sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
$Proxy160.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.remoting.rmi.RmiClientInterceptorUtils.invokeRemoteMethod(RmiClientInterceptorUtils.java:111)
org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:99)
org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.invokeInContext(AbstractRemoteSlsbInvokerInterceptor.java:141)
org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.invoke(AbstractSlsbInvokerInterceptor.java:189)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody2(RemoteStatelessEjbProxyFactoryBean.java:171)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody3$advice(RemoteStatelessEjbProxyFactoryBean.java:57)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke(RemoteStatelessEjbProxyFactoryBean.java:1)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
$Proxy253.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor612.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
java.lang.Thread.run(Thread.java:662)
]]></p></stack-trace>
</exception>
at de.mpg.escidoc.services.pubman.exceptions.ExceptionHandler.handleException(ExceptionHandler.java:93)
AxisFault: <exception>
<title><h1>500 Internal Application Server Error</h1></title>
<message><p>HTTP connection to "http://localhost:8080/fedoragsearch/rest?operation=updateIndex&action=deletePid&repositoryName=escidocrepository&indexName=content_model_admin&value=escidoc:1594776:LV" failed: <html><head><title>JBossWeb/2.0.1.GA - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>dk.defxws.fedoragsearch.server.errors.ConfigException: get stylesheet from url search/config/rest/:
; nested exception is:
dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:97)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
</pre></p><p><b>root cause</b> <pre>dk.defxws.fedoragsearch.server.errors.ConfigException: /search/config/rest/.xslt not found
dk.defxws.fedoragsearch.server.GTransformer.getTransformer(GTransformer.java:69)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:126)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:119)
dk.defxws.fedoragsearch.server.GTransformer.transform(GTransformer.java:189)
dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:154)
javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the JBossWeb/2.0.1.GA logs.</u></p><HR size="1" noshade="noshade"><h3>JBossWeb/2.0.1.GA</h3></body></html>
</p></message>
<class><p>de.escidoc.core.common.exceptions.system.ApplicationServerSystemException</p></class>
<stack-trace><p><![CDATA[
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody2(GsearchHandler.java:205)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion_aroundBody3$advice(GsearchHandler.java:57)
de.escidoc.core.common.business.indexing.GsearchHandler.requestDeletion(GsearchHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:344)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.deleteResource(IndexingHandler.java:243)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody2(IndexingHandler.java:174)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted_aroundBody3$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceDeleted(IndexingHandler.java:1)
de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemDeleted(ItemResourceListener.java:171)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete_aroundBody10(FedoraItemHandler.java:479)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure11.run(FedoraItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.delete(FedoraItemHandler.java:474)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$EnhancerByCGLIB$$f3ee31b5.delete(<generated>)
de.escidoc.core.om.service.ItemHandler.delete_aroundBody4(ItemHandler.java:123)
de.escidoc.core.om.service.ItemHandler$AjcClosure5.run(ItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.service.ItemHandler.delete(ItemHandler.java:119)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody32(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure33.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.aa.security.aop.SecurityInterceptor.proceed(SecurityInterceptor.java:237)
de.escidoc.core.aa.security.aop.SecurityInterceptor.ajc$inlineAccessMethod$de_escidoc_core_aa_security_aop_SecurityInterceptor$de_escidoc_core_aa_security_aop_SecurityInterceptor$proceed(SecurityInterceptor.java:1)
de.escidoc.core.aa.security.aop.SecurityInterceptor.authorize(SecurityInterceptor.java:206)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody34(ItemHandlerBean.java:1)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure35.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.StatisticInterceptor.proceed(StatisticInterceptor.java:205)
de.escidoc.core.common.util.aop.StatisticInterceptor.ajc$inlineAccessMethod$de_escidoc_core_common_util_aop_StatisticInterceptor$de_escidoc_core_common_util_aop_StatisticInterceptor$proceed(StatisticInterceptor.java:1)
de.escidoc.core.common.util.aop.StatisticInterceptor.createStatisticRecord(StatisticInterceptor.java:145)
de.escidoc.core.om.ejb.ItemHandlerBean.delete_aroundBody36(ItemHandlerBean.java:154)
de.escidoc.core.om.ejb.ItemHandlerBean$AjcClosure37.run(ItemHandlerBean.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.ejb.ItemHandlerBean.delete(ItemHandlerBean.java:144)
sun.reflect.GeneratedMethodAccessor614.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.invocation.Invocation.performCall(Invocation.java:359)
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
org.jboss.ejb.Container.invoke(Container.java:960)
sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
$Proxy160.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.remoting.rmi.RmiClientInterceptorUtils.invokeRemoteMethod(RmiClientInterceptorUtils.java:111)
org.springframework.ejb.access.SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:99)
org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.invokeInContext(AbstractRemoteSlsbInvokerInterceptor.java:141)
org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.invoke(AbstractSlsbInvokerInterceptor.java:189)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody2(RemoteStatelessEjbProxyFactoryBean.java:171)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke_aroundBody3$advice(RemoteStatelessEjbProxyFactoryBean.java:57)
de.escidoc.core.common.util.service.RemoteStatelessEjbProxyFactoryBean.invoke(RemoteStatelessEjbProxyFactoryBean.java:1)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
$Proxy253.delete(Unknown Source)
sun.reflect.GeneratedMethodAccessor612.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
java.lang.Thread.run(Thread.java:662)
]]></p></stack-trace>
</exception>
at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
Experimente[edit]
- fgsindex.defaultWriteLockTimeout = 3000 (msec anstelle von 100.000.000) wait time for an IndexWriter for the same directory in case of a write.lock before throwing a LockObtainFailedException
de.mpg.escidoc.services.common.exceptions.TechnicalException: <exception>
<title><h1>500 Internal eSciDoc System Error</h1></title>
<message><p><?xml version="1.0" encoding="UTF-8"?>
<resultPage>
<error>
<message>Update error pidOrFilename=escidoc:1602273; nested exception is:
dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexWriter new error, creating index indexName=item_container_admin :
; nested exception is:
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/coreservice/server/default/data/index/lucene/item_container_admin/write.lock</message>
</error>
</resultPage>
</p></message>
<class><p>de.escidoc.core.common.exceptions.system.SystemException</p></class>
<stack-trace><p><![CDATA[
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody8(IndexingHandler.java:502)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody9$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody6(IndexingHandler.java:338)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing_aroundBody7$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.doIndexing(IndexingHandler.java:1)
de.escidoc.core.common.business.indexing.IndexingHandler.indexResource(IndexingHandler.java:285)
de.escidoc.core.common.business.indexing.IndexingHandler.addResource(IndexingHandler.java:228)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated_aroundBody0(IndexingHandler.java:153)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated_aroundBody1$advice(IndexingHandler.java:57)
de.escidoc.core.common.business.indexing.IndexingHandler.resourceCreated(IndexingHandler.java:1)
de.escidoc.core.om.business.fedora.item.ItemResourceListener.fireItemCreated(ItemResourceListener.java:98)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.create_aroundBody6(FedoraItemHandler.java:434)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$AjcClosure7.run(FedoraItemHandler.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler.create(FedoraItemHandler.java:409)
de.escidoc.core.om.business.fedora.item.FedoraItemHandler$$FastClassByCGLIB$$575498c2.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke_aroundBody2(PerformanceStatisticsInterceptor.java:89)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor$AjcClosure3.run(PerformanceStatisticsInterceptor.java:1)
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
de.escidoc.core.common.util.aop.TraceInterceptor.traceMethod(TraceInterceptor.java:57)
de.escidoc.core.om.performance.PerformanceStatisticsInterceptor.invoke(PerformanceStatisticsInterceptor.java:80)
- project FgsLucene OperationsImpl.java ohne closeIndexReader
ERROR 2013-02-12 08:35:14,415 [http-0.0.0.0-8080-8] (RESTImpl) RESTImpl error:
dk.defxws.fedoragsearch.server.errors.GenericSearchException: IndexReader open error indexName=item_container_admin :
; nested exception is:
java.io.FileNotFoundException: /data/coreservice/server/default/data/index/lucene/item_container_admin/_7690.prx (Too many open files)
at dk.defxws.fgslucene.OperationsImpl.getIndexReader(OperationsImpl.java:679)
at dk.defxws.fgslucene.OperationsImpl.updateIndex(OperationsImpl.java:265)
at dk.defxws.fedoragsearch.server.GenericOperationsImpl.updateIndex(GenericOperationsImpl.java:356)
at dk.defxws.fedoragsearch.server.RESTImpl.updateIndex(RESTImpl.java:280)
at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:117)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:662)