Member since
06-09-2016
12
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2625 | 10-29-2018 10:09 AM |
10-29-2018
10:09 AM
Let me share my workflow to support anybody hitting this in the future. Many thanks again Jay! [mmartofel@emlpsn01 certs]$ sudo openssl genrsa -passout pass:hadoop -out $AMBARI_SERVER_HOSTNAME.key 2048
Generating RSA private key, 2048 bit long modulus
.........................................+++
...............................................................+++
e is 65537 (0x10001)
[mmartofel@emlpsn01 certs]$ ll
total 4
-rw-r--r-- 1 root root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo chown ambari *
[mmartofel@emlpsn01 certs]$ ll
total 4
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo openssl req -new -key $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.csr -subj "/C=DE/ST=Bavaria/L=Minich/O=EMNOS/CN=$AMBARI_SERVER_HOSTNAME"
[mmartofel@emlpsn01 certs]$ ll
total 8
-rw-r--r-- 1 root root 1001 Oct 27 14:50 emlpsn01.emprd.lpemrz.com.csr
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo openssl x509 -req -days 1365 -in $AMBARI_SERVER_HOSTNAME.csr -signkey $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.crt
Signature ok
subject=/C=DE/ST=Bavaria/L=Minich/O=EMNOS/CN=emlpsn01.emprd.lpemrz.com
Getting Private key
[mmartofel@emlpsn01 certs]$ ll
total 12
-rw-r--r-- 1 root root 1192 Oct 27 14:51 emlpsn01.emprd.lpemrz.com.crt
-rw-r--r-- 1 root root 1001 Oct 27 14:50 emlpsn01.emprd.lpemrz.com.csr
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ pwd
/etc/ambari-server/certs
[mmartofel@emlpsn01 certs]$ cd ..
[mmartofel@emlpsn01 ambari-server]$ cd conf
[mmartofel@emlpsn01 conf]$ ls -al
total 48
drwxr-xr-x 3 ambari root 248 Oct 27 14:35 .
drwxr-xr-x 7 ambari root 127 Oct 27 14:44 ..
-rwxr-xr-x 1 ambari root 8533 Oct 27 14:40 ambari.properties
-rwxr-xr-x 1 ambari root 7900 Oct 26 10:36 ambari.properties.rpmsave.20181026104227
-rwxr-xr-x 1 ambari root 317 Oct 26 18:24 krb5JAASLogin.conf
-rw-r--r-- 1 ambari ambari 317 Oct 26 18:24 krb5JAASLogin.conf.bak
-rw-r----- 1 ambari root 21 Aug 22 14:59 ldap-password.dat
-rwxr-xr-x 1 ambari root 4929 Oct 26 10:36 log4j.properties
-rwxr-xr-x 1 ambari root 2630 May 29 21:34 metrics.properties
-rw-r----- 1 ambari root 12 Jun 8 16:23 password.dat
drwxr-xr-x 2 ambari root 6 Oct 27 14:35 truststore
[mmartofel@emlpsn01 truststore]$ cd ..
[mmartofel@emlpsn01 conf]$ ls
ambari.properties ambari.properties.rpmsave.20181026104227 krb5JAASLogin.conf krb5JAASLogin.conf.bak ldap-password.dat log4j.properties metrics.properties password.dat truststore
[mmartofel@emlpsn01 conf]$ pwd
/etc/ambari-server/conf
[mmartofel@emlpsn01 conf]$ sudo mv ./truststore/ /etc/ambari-server/
[mmartofel@emlpsn01 conf]$ ls
ambari.properties ambari.properties.rpmsave.20181026104227 krb5JAASLogin.conf krb5JAASLogin.conf.bak ldap-password.dat log4j.properties metrics.properties password.dat
[mmartofel@emlpsn01 conf]$ ll
total 48
-rwxr-xr-x 1 ambari root 8533 Oct 27 14:40 ambari.properties
-rwxr-xr-x 1 ambari root 7900 Oct 26 10:36 ambari.properties.rpmsave.20181026104227
-rwxr-xr-x 1 ambari root 317 Oct 26 18:24 krb5JAASLogin.conf
-rw-r--r-- 1 ambari ambari 317 Oct 26 18:24 krb5JAASLogin.conf.bak
-rw-r----- 1 ambari root 21 Aug 22 14:59 ldap-password.dat
-rwxr-xr-x 1 ambari root 4929 Oct 26 10:36 log4j.properties
-rwxr-xr-x 1 ambari root 2630 May 29 21:34 metrics.properties
-rw-r----- 1 ambari root 12 Jun 8 16:23 password.dat
[mmartofel@emlpsn01 conf]$ cd ..
[mmartofel@emlpsn01 ambari-server]$ ll
total 0
drwxr-xr-x 2 ambari root 117 Oct 27 14:51 certs
drwxr-xr-x 2 ambari root 230 Oct 27 14:56 conf
drwxr-xr-x 2 ambari root 131 Jun 8 15:22 conf_08_06_18_16_15.save
drwxr-xr-x 2 ambari root 127 Apr 11 2018 conf_12_04_18_10_36.save
drwxr-xr-x 2 ambari root 101 Apr 12 2018 conf_12_04_18_11_25.save
drwxr-xr-x 2 ambari root 6 Oct 27 14:35 truststore
[mmartofel@emlpsn01 truststore]$ pwd
/etc/ambari-server/truststore
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)? y
The truststore is already configured. Do you want to re-configure the truststore [y/n] (y)? y
TrustStore type [jks/jceks/pkcs12] (jks):
Path to TrustStore file :/etc/ambari-server/truststore
Password for TrustStore:
Re-enter password:
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$ pwd
/etc/ambari-server/truststore
[mmartofel@emlpsn01 truststore]$ ll
total 0
[mmartofel@emlpsn01 truststore]$ sudo keytool -import -file /etc/ambari-server/certs/emlpsn01.emprd.lpemrz.com.crt -alias ambari-server -keystore ambari-server-truststore.jks
Enter keystore password:
Re-enter new password:
Owner: CN=emlpsn01.emprd.lpemrz.com, O=EMNOS, L=Minich, ST=Bavaria, C=DE
Issuer: CN=emlpsn01.emprd.lpemrz.com, O=EMNOS, L=Minich, ST=Bavaria, C=DE
Serial number: d2977919873473e6
Valid from: Sat Oct 27 14:51:04 CEST 2018 until: Sat Jul 23 14:51:04 CEST 2022
Certificate fingerprints:
MD5: CA:F2:C2:60:CF:73:81:6C:C9:B8:E6:69:B7:CB:CE:D0
SHA1: CA:F7:E0:B6:68:C3:C7:6B:DC:49:3A:10:3C:93:8A:28:52:B2:C2:D6
SHA256: B3:50:84:3A:AB:B5:84:0D:A7:8F:0F:12:BC:6D:4B:C4:51:13:E0:A6:D0:CD:F9:A5:A6:E4:72:6D:E6:FF:A8:1C
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 1
Trust this certificate? [no]: yes
Certificate was added to keystore
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 root root 910 Oct 27 15:03 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo chown ambari ./*
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 ambari root 910 Oct 27 15:03 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)?
The truststore is already configured. Do you want to re-configure the truststore [y/n] (y)?
TrustStore type [jks/jceks/pkcs12] (jks):
Path to TrustStore file :/etc/ambari-server/truststore/ambari-server-truststore.jks
Password for TrustStore:
Re-enter password:
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$
[mmartofel@emlpsn01 truststore]$
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 5
Do you want to configure a truststore [y/n] (y)?
Do you want to import a certificate [y/n] (y)?
Please enter an alias for the certificate: ambari-server
Enter path to certificate: /etc/ambari-server/certs/emlpsn01.emprd.lpemrz.com.crt
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 ambari root 910 Oct 27 15:06 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo keytool --list --keystore ./ambari-server-truststore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
ambari-server, Oct 27, 2018, trustedCertEntry,
Certificate fingerprint (SHA1): CA:F7:E0:B6:68:C3:C7:6B:DC:49:3A:10:3C:93:8A:28:52:B2:C2:D6
[mmartofel@emlpsn01 truststore]$ sudo ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
[mmartofel@emlpsn01 truststore]$ cat /etc/ambari-server/conf/ambari.properties | grep trust
kerberos.operation.verify.kdc.trust=true
ssl.trustStore.password=XXXXXXXXXXXXXXX
ssl.trustStore.path=/etc/ambari-server/truststore/ambari-server-truststore.jks
ssl.trustStore.type=jks
... View more
10-28-2018
05:22 PM
Have only one line for Kerberos: [mmartofel@emlpsn01 conf]$ grep 'trust' /etc/ambari-server/conf/ambari.properties
kerberos.operation.verify.kdc.trust=true
[mmartofel@emlpsn01 conf]$ grep 'trust' /etc/ambari-server/conf/ambari.properties.rpmsave.20181026104227
[mmartofel@emlpsn01 conf]$
... View more
10-27-2018
01:15 PM
YES! This works now! Many thanks for your prompt support Jay! Will sum up my steps later on for the next folks hitting this issue.
... View more
10-27-2018
08:19 AM
I set same password for admin and amb_ranger_admin as of the instructions from: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Ranger_Install_Guide/content/updating_ranger_admin_passwords.html and https://community.hortonworks.com/questions/19948/this-alert-is-used-to-ensure-that-the-ranger-admin.html Also I created new truststore file as of article: https://community.hortonworks.com/articles/16373/ranger-ssl-pitfalls.html There is many truststore files and accompaining passwords along Ranger but can't find which one really is considered by upgrade pre check process. Could you please point me to correct on? Any more tracing, debugging I can do here?
... View more
10-27-2018
08:14 AM
Upgrading 2.5.3 to 2.6.2 , actual Ambari is 2.6.2.2 Last pre check left to resolve. Ranger complains at ambari-server.log : 27 Oct 2018 02:41:41,564 INFO [ambari-client-thread-55] RangerSSLConfigCheck:72 - Ranger is SSL enabled, need to show Configuration changes warning before upragade proceeds.
27 Oct 2018 02:42:43,084 ERROR [ambari-client-thread-52] URLStreamProvider:297 - Can't get secure connection to https://emlpsn01.emprd.lpemrz.com:6182/service/public/api/repository/count. Truststore path or password is not set.
27 Oct 2018 02:42:43,085 ERROR [ambari-client-thread-52] CheckHelper:109 - Check SERVICES_RANGER_PASSWORD_VERIFY failed
java.lang.IllegalStateException: Can't get secure connection to https://emlpsn01.emprd.lpemrz.com:6182/service/public/api/repository/count. Truststore path or password is not set.
at org.apache.ambari.server.controller.internal.URLStreamProvider.getSSLConnection(URLStreamProvider.java:298)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:181)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:160)
at org.apache.ambari.server.checks.RangerPasswordCheck.checkLogin(RangerPasswordCheck.java:243)
at org.apache.ambari.server.checks.RangerPasswordCheck.perform(RangerPasswordCheck.java:132)
at org.apache.ambari.server.state.CheckHelper.performChecks(CheckHelper.java:104)
... View more
Labels:
10-22-2018
01:38 PM
2 Kudos
Enable System Service Mode On an Upgraded Cluster This solves the problem. Had the sam issue.
... View more
10-22-2018
01:34 PM
Many thanks Cibi! It works now. Indeed yarn-system queue was with 0% capacity and is_hbase_system_service_launch was false.
... View more
10-20-2018
01:52 AM
Myself I do not have any issues with class not found but: client.ConnectionImplementation: Retrieve cluster id failed
may be any related so posting 018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.0.1.0-187/hadoop/lib/native
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-514.21.1.el7.x86_64
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.name=yarn-ats
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.home=/home/yarn-ats
2018-10-19 22:48:33,152 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.dir=/home/yarn-ats
2018-10-19 22:48:33,154 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Initiating client connection, connectString=emltgh01.emtst.lpemrz.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/966280619@62068d2d
2018-10-19 22:48:33,171 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server emltgh01.emtst.lpemrz.com/10.10.13.100:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-19 22:48:33,174 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.10.13.100:38167, server: emltgh01.emtst.lpemrz.com/10.10.13.100:2181
2018-10-19 22:48:33,179 INFO [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server emltgh01.emtst.lpemrz.com/10.10.13.100:2181, sessionid = 0x1668d9e70b70055, negotiated timeout = 40000
2018-10-19 22:48:33,188 WARN [main] client.ConnectionImplementation: Retrieve cluster id failed
java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:527)
at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:301)
at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277)
at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323)
at java.lang.Thread.run(Thread.java:745)
... View more