Support Questions

Find answers, ask questions, and share your expertise

org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException occurs occasionally

New Contributor

Occasionally on our system, we get the follow error. And was looking to find any workaround or if it is a known/fixed issue?

java.util.concurrent.ExecutionException: org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException: ERROR 900 (HJ01): Hash Join cache not found joinId: -4640437492055623646. The cache might have expired and have been removed.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:900)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:836)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
	at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
	at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
	at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
	at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
	at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
	at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.getCurrenciesFromResultSet(AccountsDaoImpl.java:771)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.collectData(AccountsDaoImpl.java:849)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.getCurrenciesByEffectiveDate(AccountsDaoImpl.java:302)
	at com.eagleinvsys.organization.service.CurrencyService.getCurrenciesByEffectiveDate(CurrencyService.java:95)
	at com.eagleinvsys.organization.service.CurrencyService$FastClassBySpringCGLIB$44f0260b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
	at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:345)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:408)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)
	at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.eagleinvsys.organization.service.CurrencyService$EnhancerBySpringCGLIB$8fdd8ad3.getCurrenciesByEffectiveDate(<generated>)
	at com.eagleinvsys.organization.controller.CurrencyController.getCurrenciesByEffectiveDate(CurrencyController.java:165)
	at com.eagleinvsys.organization.controller.CurrencyController$FastClassBySpringCGLIB$fed0a43f.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:139)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.eagleinvsys.organization.controller.CurrencyController$EnhancerBySpringCGLIB$4b3ccf09.getCurrenciesByEffectiveDate(<generated>)
	at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:169)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException: ERROR 900 (HJ01): Hash Join cache not found joinId: -4640437492055623646. The cache might have expired and have been removed.
	at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:139)
	at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
	at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:250)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:111)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 common frames omitted
1 ACCEPTED SOLUTION

Contributor

you may want to increase value of

phoenix.coprocessor.maxServerCacheTimeToLiveMs
Maximum living time (in milliseconds) of server caches. A cache entry expires after this amount of time has passed since last access. Consider adjusting this parameter when a server-side IOException(“Could not find hash cache for joinId”) happens. Getting warnings like “Earlier hash cache(s) might have expired on servers” might also be a sign that this number should be increased.

http://phoenix.apache.org/tuning.html

View solution in original post

6 REPLIES 6

Contributor

you may want to increase value of

phoenix.coprocessor.maxServerCacheTimeToLiveMs
Maximum living time (in milliseconds) of server caches. A cache entry expires after this amount of time has passed since last access. Consider adjusting this parameter when a server-side IOException(“Could not find hash cache for joinId”) happens. Getting warnings like “Earlier hash cache(s) might have expired on servers” might also be a sign that this number should be increased.

http://phoenix.apache.org/tuning.html

New Contributor

Thanks! Will give this a try and see if it helps.

New Contributor

We made this change above, basically tripling the default value. So far haven't seen the original issue again, but we do get a new warning message:

org.apache.phoenix.cache.ServerCacheClient
Unable to remove hash cache for [region=xxxx, hostname=xxxxx,seqNum=2, region=xxxx, hostname=xxxx, ...]

Contributor

good to know you solved the original problem. not sure what the warning means though. Maybe someone from community can help

New Contributor

Spoke too soon, we do see the original issue still as well. 😞

Contributor

in that case you would want to further increase cache timeout or break your query into multiple queries and use temporary tables.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.