org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException occurs occasionally

[#1]

Occasionally on our system, we get the follow error. And was looking to find any workaround or if it is a known/fixed issue?

java.util.concurrent.ExecutionException: org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException: ERROR 900 (HJ01): Hash Join cache not found joinId: -4640437492055623646. The cache might have expired and have been removed.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:900)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:836)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:812)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:719)
	at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
	at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
	at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
	at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
	at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
	at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.getCurrenciesFromResultSet(AccountsDaoImpl.java:771)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.collectData(AccountsDaoImpl.java:849)
	at com.eagleinvsys.organization.dao.AccountsDaoImpl.getCurrenciesByEffectiveDate(AccountsDaoImpl.java:302)
	at com.eagleinvsys.organization.service.CurrencyService.getCurrenciesByEffectiveDate(CurrencyService.java:95)
	at com.eagleinvsys.organization.service.CurrencyService$FastClassBySpringCGLIB$44f0260b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
	at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:345)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:408)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)
	at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.eagleinvsys.organization.service.CurrencyService$EnhancerBySpringCGLIB$8fdd8ad3.getCurrenciesByEffectiveDate(<generated>)
	at com.eagleinvsys.organization.controller.CurrencyController.getCurrenciesByEffectiveDate(CurrencyController.java:165)
	at com.eagleinvsys.organization.controller.CurrencyController$FastClassBySpringCGLIB$fed0a43f.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:139)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.eagleinvsys.organization.controller.CurrencyController$EnhancerBySpringCGLIB$4b3ccf09.getCurrenciesByEffectiveDate(<generated>)
	at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:169)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.coprocessor.HashJoinCacheNotFoundException: ERROR 900 (HJ01): Hash Join cache not found joinId: -4640437492055623646. The cache might have expired and have been removed.
	at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:139)
	at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:125)
	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
	at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:250)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:111)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:106)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 common frames omitted

[#2 · Accepted solution]

you may want to increase value of

phoenix.coprocessor.maxServerCacheTimeToLiveMs

Maximum living time (in milliseconds) of server caches. A cache entry expires after this amount of time has passed since last access. Consider adjusting this parameter when a server-side IOException(“Could not find hash cache for joinId”) happens. Getting warnings like “Earlier hash cache(s) might have expired on servers” might also be a sign that this number should be increased.

http://phoenix.apache.org/tuning.html

[#3]

Thanks! Will give this a try and see if it helps.

[#4]

We made this change above, basically tripling the default value. So far haven't seen the original issue again, but we do get a new warning message:

org.apache.phoenix.cache.ServerCacheClient
Unable to remove hash cache for [region=xxxx, hostname=xxxxx,seqNum=2, region=xxxx, hostname=xxxx, ...]

[#5]

good to know you solved the original problem. not sure what the warning means though. Maybe someone from community can help

[#6]

Spoke too soon, we do see the original issue still as well. 😞

[#7]

in that case you would want to further increase cache timeout or break your query into multiple queries and use temporary tables.