Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH 5.15.1 upgrade java exception

CDH 5.15.1 upgrade java exception

New Contributor

Hello!

 

I'm trying to upgrade the CDH on my cluster from 5.15.0 to 5.15.1. I've already upgraded the CM to 5.15.1. After I distributed the latest CDH parcel and push the 'Upgrade' button, I got this java exception:

2018-08-28 10:33:38,146 INFO 200544445@scm-web-71:com.cloudera.server.web.common.ExceptionReport: Exception report generated accessing http://xxxxx:7180/cmf/clusters/2/upgradeWizard
java.lang.StringIndexOutOfBoundsException: String index out of range: -12
        at java.lang.String.substring(String.java:1967)
        at com.cloudera.cmf.model.DbHostHeartbeat.checkJavaComponent(DbHostHeartbeat.java:177)
        at com.cloudera.cmf.model.DbHostHeartbeat.getActiveComponentInfo(DbHostHeartbeat.java:207)
        at com.cloudera.cmf.service.components.ReleaseDetector$ReleaseInfo.add(ReleaseDetector.java:202)
        at com.cloudera.cmf.service.components.ReleaseDetector$ReleaseInfo.access$600(ReleaseDetector.java:119)
        at com.cloudera.cmf.service.components.ReleaseDetector.detect(ReleaseDetector.java:350)
        at com.cloudera.cmf.service.components.ReleaseDetector.detect(ReleaseDetector.java:454)
        at com.cloudera.server.web.cmf.wizard.cluster.ClusterUpgradeController.getClusterUpgradeWizard(ClusterUpgradeController.java:159)
        at com.cloudera.server.web.cmf.wizard.cluster.ClusterUpgradeController$$FastClassByCGLIB$$9b421e20.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        at com.cloudera.server.web.cmf.wizard.cluster.ClusterUpgradeController$$EnhancerByCGLIB$$b066fc80.getClusterUpgradeWizard(<generated>)
        at sun.reflect.GeneratedMethodAccessor1917.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

 

 

Does anyone have any ideas, why it throws this exception?

 

Thank you!

8 REPLIES 8

Re: CDH 5.15.1 upgrade java exception

New Contributor

Hi All,

the same StringIndexOutOfBoundsException came up to me too. The 5.15.0 version is full of bug while the upgrade from 5.15.0 to 5.15.1 is not working so we are keen to know how to solve the situation.
Any ideas are welcomed!

Re: CDH 5.15.1 upgrade java exception

New Contributor

We ended up downgrading the installed packages back to 5.15.0. I had a team of 3 people working on this for 2 days. The class where the error occurs appears to be Cloudera-proprietary. We could not find it online anywhere.

 

We would love to see a real solution from Cloudera for this problem. As of now, we have no upgrade path off of 5.15.0.

 

Here are some resources we ended up using. The upgrade from 5.15.0 to 5.15.1 does not appear to have changed the database schema (at least in our Postgres instance), so we dodged a bullet there.

 

https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_cm_downgrade.html

 

https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/pool/contrib/e/enterprise/

Re: CDH 5.15.1 upgrade java exception

Super Guru

@jason-p2r, @amakoviczki, and @zokis,

 

I'll alert Cloudera Engineering and open an internal Cloudera Jira to look at this issue.

We did introduce some new code in 5.15.1 to look at Java version and this is where you are seeing the failure.

 

It would help if we knew what is being parsed so we can get a better idea of what may be going wrong.

 

Please post your "java -version" output on the Cloudera Manager host.  It may give us some clues.

 

I'll keep this thread updated on progress.

 

Sorry for the troubles and thank you for bringing it up so we can address the problem.

 

-Ben

Highlighted

Re: CDH 5.15.1 upgrade java exception

Cloudera Employee

Please give details about the Java version being run on the hosts. Cloudera Manager is having an issue reading the version output from the Java installed on the host.

Re: CDH 5.15.1 upgrade java exception

New Contributor

@bgooley, @Chris Davis

 

We use this java version:

 

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

 

This is supported by Cloudera. (https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#conc...)

 

Thanks for your help!

Re: CDH 5.15.1 upgrade java exception

Super Guru

@amakoviczki,

 

The assumption when we asked for your java version was that you had the same java version in your path that was set for JAVA_HOME across all hosts in your cluster.  I was able to reproduce the exception if I had an older Cloudera Agent heartbeating into a 5.12.1 Cloudera Manager AND it detected OpenJDK as one of the JDKs installed.

 

Can you run a host inspector report and see if any OpenJDK versions are detected?

 

As mentioned, I could reproduce, but only if I also had an older agent heartbeating, too.  Make certain you have upgraded all the cluster's hosts to 5.15.1 agents and the agents were restarted before upgrading CDH.

 

Thank you for your help as we attempt to verify the cause.  We really appeciate it.

Re: CDH 5.15.1 upgrade java exception

Super Guru

@amakoviczki, @jason-p2r, @zokis,

 

I have reproduced and we believe we know what the cause is:

 

- Cloudera Manager is at version 5.15.1

- One or more agents heartbeating to CM has an agent package at a version before 5.15.1

- That version of java reported is OpenJDK.

 

We'll look at addressing the issue in future releases, but, for now, depending on your situation, intended configuration one of the following may help:

 

- Ensure that all agents find an Oracle JDK

- Ensure that all agents have been upgraded to 5.15.1 before attempting CDH upgrade

- use CM 5.15.0 to upgrade to CDH 5.15.1 (we haven't tested, but I don't see why that won't work since the offending code is not in 5.15.0). NOTE:  it is supported to run CM 5.15.0 with CDH 5.15.1 since it is a maintenance release.

 

NOTE:

 

Curently, Cloudera only supports and tests on Oracle JDKs.  This is likely to change in the future, but we are still evaluating OpenJDK support.  This means we only test with Oracle JDK and thus did not find the issue you reported that seems to involve a non-Oracle JDK

See:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_...

 

NOTE2:

 

I mentioned that CM 5.15.0 can be used to manage/upgrade CDH to 5.15.1.  This is the documentation that states this:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#cm_c...

 

Re: CDH 5.15.1 upgrade java exception

New Contributor

Thanks for the quick troubleshooting. It is quite possible that OpenJDK found its way into a path or two. We always allow Cloudera to install and manage the JDK it uses when we install, but OpenJDK may also be installed onto some of the servers. Thanks again for the thorough answer.

Don't have an account?
Coming from Hortonworks? Activate your account here