Support Questions
Find answers, ask questions, and share your expertise

Migrating from Oracle JDK to Open JDK (CDH 6.3.0)

Explorer

Testing this on our non-prod cluster and finding Cloudera manager won't start after switching from Oracle to Open JDK. Followed the Cloudera documentation and it appears to be working, but after removing Oracle RPM things went kablooey. Anyone have any suggestions? From CM log:

 

cResourcePool$ScatteredAcquireTask@1f258f49 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201)
at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:103)
at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:428)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186)

1 ACCEPTED SOLUTION

Master Collaborator

It could be. Check the file $JAVA_HOME/jre/lib/security/java.security. You'll find something like the below in it:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
    include jdk.disabled.namedCurves

 

If TLSv1 and TLSv1.1 are disabled in your version, you can try to remove them from the disabled list and try again.

 

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

19 REPLIES 19

Master Collaborator

@OptOut ,

 

Could you please provide the link to the doc you followed?

 

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Master Collaborator

What's your OpenJDK version?

Could you please share the contents of your /etc/default/cloudera-scm-server file?

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

java-1.8.0-openjdk - version 1:1.8.0.322.b06-1.el7_9

 

/etc/default/cloudera-scm-server :

#
# Specify any command line arguments for the Cloudera SCM Server here.
#

CMF_SERVER_ARGS=""

#
# Locate the JDBC driver jar file.
#
# The default value is the default system mysql driver on RHEL/CentOS/Ubuntu
# and the standard, documented location for where to put the oracle jar in CM
# deployments.
#

export CMF_JDBC_DRIVER_JAR="/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar"

#
# Customize the TLS ciphers used by Cloudera Manager.
#
# Cloudera Manager uses the Modern list of TLS ciphers, as defined by Mozilla
# (reference: https://wiki.mozilla.org/Security/Server_Side_TLS).
# Some older JDK versions might not have support for newer ciphers in that list,
# potentially leading to failures to connect to the Admin Console when TLS is
# enabled. You can customize the set of ciphers by uncommenting and editing
# the line below. The example provided will set the list of ciphers to the
# Intermediate list defined by Mozilla.
#
#export CMF_OVERRIDE_TLS_CIPHERS="TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_DHE_RSA_WITH_AES_128_GCM_SHA256:TLS_DHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_AES_256_CBC_SHA256:TLS_DHE_RSA_WITH_AES_256_CBC_SHA:TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_EDH_RSA_WITH_3DES_EDE_CBC_SHA:TLS_RSA_WITH_AES_128_GCM_SHA256:TLS_RSA_WITH_AES_256_GCM_SHA384:TLS_RSA_WITH_AES_128_CBC_SHA256:TLS_RSA_WITH_AES_256_CBC_SHA256:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:TLS_RSA_WITH_3DES_EDE_CBC_SHA"

#
# Java Options.
#
# Default value sets Java maximum heap size to 2GB, and Java maximum permanent
# generation size to 256MB.
#

export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

 

Explorer

Made some progress since this morning.

Rebooted the cluster and that allowed me to log in to CM. CM was showing Open JDK as would be expected (under support/about). 

Saw an error in the log and figured out that CM had path to Oracle JDK cert store in parameters. Removed that and CM service restarted successfully..

Attempted rolling restart of services to get cluster back up and running. HDFS and a few others came up, but restart stopped at Sentry with following: 

sentry.sh

at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201)
at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
... 39 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:103)
at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:428)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186)
... 58 more

 

If someone could point me in the right direction or advise I would appreciate it. From Cloudera documentation, I should not have to set TLS protocols and they are commented out in server config file.

Explorer

Digging into this a little more, I see the old Oracle JDK location in a few config settings. Can someone tell me if it is safe to just remove this and restart CM? Or do I need to build truststore in Open JDK location? 

Master Collaborator

@OptOut ,

 

I think you missed a step from the documentation page you referenced.

In step 3 it says:

 

"3. On the Cloudera Manager Server host only (not required for other hosts):


a. Open the file /etc/default/cloudera-scm-server in a text editor.
b. Edit the line that begins with export JAVA_HOME (if this line does not exist, add it) and change the path to the path of the new JDK (the JDK is usually installed in /usr/lib/jvm)(or /usr/lib64/jvm on SLES 12), but the path may differ depending on how the JDK was installed)."

This doesn't seem to have been done in your case.

 

Please re-check the documentation and make sure that all the steps are covered.

 

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

With that in the settings, CM does not start. I think it is because this is version 6.3.0 and perhaps that setting is not valid until a minor version higher. Do you know if I can remove the custom settings pointing to the old trusstore (/usr/java/default/jre/lib/security/jssecacerts ) as this directory no longer exists?

Explorer

I'll give this a try, anyway. I don't think I had it in the current state the other time I tried it. Will post results.

Master Collaborator

If it fails, please share the modified setting so that I can take a look.

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Added this setting:

export JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk"

Restart of CM worked this time.

Rolling restart of services fails with same error above at Sentry service.

Do you know if it is safe to remove custom keystore settings for oracle jdk? I can't find a lot of info on this. This is a Kerberized cluster.

 

Looks like the setting is fine from service status: cm-server[135721]: JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk

Master Collaborator

Now that CM has started correctly, go to Hosts > All Hosts > Configuration and set the "Java Home Directory" property to /usr/lib/jvm/java-1.8.0-openjdk

 

After that do a full cluster restart from CM.

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Applied the config, but get same error from Sentry as above when initiating Rolling Restart to apply configuration.

Master Collaborator

Is SSL enabled for your MySQL?

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Not sure how to check for that.

Explorer

I suspect it is. I have a working theory now. The mysql version installed looks like it supports TLS 1.0 and 1.1 only. This is a recent version of Open JDK and it could be that it does not work with TLS below 1.2.

Master Collaborator

It could be. Check the file $JAVA_HOME/jre/lib/security/java.security. You'll find something like the below in it:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
    include jdk.disabled.namedCurves

 

If TLSv1 and TLSv1.1 are disabled in your version, you can try to remove them from the disabled list and try again.

 

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Thanks for the reply! I had come to the same conclusion, although you may not believe it. Sentry is back up and I'm trying to restart things to see where else it might break, at this time! I appreciate your replies and will follow up if I succeed or hit a different issue. 

Explorer

Okay, only Oozie also needed the change but now everything is working! Will have to figure out longer term plan, but first things first. 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.