question Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set. in Support Questions

2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

mmartofel — Sat, 27 Oct 2018 15:14:49 GMT

Upgrading 2.5.3 to 2.6.2 , actual Ambari is 2.6.2.2

Last pre check left to resolve. Ranger complains at ambari-server.log :

27 Oct 2018 02:41:41,564  INFO [ambari-client-thread-55] RangerSSLConfigCheck:72 - Ranger is SSL enabled, need to show Configuration changes warning before upragade proceeds.
27 Oct 2018 02:42:43,084 ERROR [ambari-client-thread-52] URLStreamProvider:297 - Can't get secure connection to https://emlpsn01.emprd.lpemrz.com:6182/service/public/api/repository/count.  Truststore path or password is not set.
27 Oct 2018 02:42:43,085 ERROR [ambari-client-thread-52] CheckHelper:109 - Check SERVICES_RANGER_PASSWORD_VERIFY failed
java.lang.IllegalStateException: Can't get secure connection to https://emlpsn01.emprd.lpemrz.com:6182/service/public/api/repository/count.  Truststore path or password is not set.
	at org.apache.ambari.server.controller.internal.URLStreamProvider.getSSLConnection(URLStreamProvider.java:298)
	at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:181)
	at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:160)
	at org.apache.ambari.server.checks.RangerPasswordCheck.checkLogin(RangerPasswordCheck.java:243)
	at org.apache.ambari.server.checks.RangerPasswordCheck.perform(RangerPasswordCheck.java:132)
	at org.apache.ambari.server.state.CheckHelper.performChecks(CheckHelper.java:104)

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

jsensharma — Sat, 27 Oct 2018 15:18:53 GMT

@Marek Martofel

Can you please check if you have setup Abari Truststore?

Do you see any 'truststore' related settings in your current or old "/etc/ambari-server/conf/ambari.properties" ?

# grep 'trust' /etc/ambari-server/conf/ambari.properties
# grep 'trust' /etc/ambari-server/conf/ambari.properties.rpmsave

Based on the error it looks like Your Ranger UI is running on HTTPs and ambari truststore does not have the Ranger certificate imported to it.

You can refer to the following doc to know more about Setting up Truststore for Ambari : https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-security/content/set_up_truststore_for_ambari_server.html

The following HCC article also explains the cause and remedy of "Truststore path or password is not set"

https://community.hortonworks.com/articles/39865/enabling-https-for-ambariserver-and-troubleshootin.html

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

mmartofel — Sat, 27 Oct 2018 15:19:39 GMT

I set same password for admin and amb_ranger_admin as of the instructions from:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Ranger_Install_Guide/content/updating_ranger_admin_passwords.html

and

https://community.hortonworks.com/questions/19948/this-alert-is-used-to-ensure-that-the-ranger-admin.html

Also I created new truststore file as of article:

https://community.hortonworks.com/articles/16373/ranger-ssl-pitfalls.html

There is many truststore files and accompaining passwords along Ranger but can't find which one really is considered by upgrade pre check process.

Could you please point me to correct on?

Any more tracing, debugging I can do here?

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

jsensharma — Sat, 27 Oct 2018 15:26:57 GMT

@Marek Martofel

So basically you should do the following:

1. Setup truststore for Ambari Server : (option-4)

# ambari-server setup-seturity
[4] Setup truststore

2. Import Ranger certificate inside the ambari truststore it can also be done manually or using the following option (option-5)

# ambari-server setup-seturity
[5] Import certificate to truststore.

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

mmartofel — Sat, 27 Oct 2018 20:15:33 GMT

YES! This works now! Many thanks for your prompt support Jay!

Will sum up my steps later on for the next folks hitting this issue.

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

mmartofel — Mon, 29 Oct 2018 00:22:51 GMT

Have only one line for Kerberos:

[mmartofel@emlpsn01 conf]$ grep 'trust' /etc/ambari-server/conf/ambari.properties
kerberos.operation.verify.kdc.trust=true
[mmartofel@emlpsn01 conf]$ grep 'trust' /etc/ambari-server/conf/ambari.properties.rpmsave.20181026104227
[mmartofel@emlpsn01 conf]$

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

mmartofel — Mon, 29 Oct 2018 17:09:31 GMT

Let me share my workflow to support anybody hitting this in the future.

Many thanks again Jay!

[mmartofel@emlpsn01 certs]$ sudo openssl genrsa -passout pass:hadoop -out $AMBARI_SERVER_HOSTNAME.key 2048
Generating RSA private key, 2048 bit long modulus
.........................................+++
...............................................................+++
e is 65537 (0x10001)
[mmartofel@emlpsn01 certs]$ ll
total 4
-rw-r--r-- 1 root root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo chown ambari *
[mmartofel@emlpsn01 certs]$ ll
total 4
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo openssl req -new -key $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.csr -subj "/C=DE/ST=Bavaria/L=Minich/O=EMNOS/CN=$AMBARI_SERVER_HOSTNAME"
[mmartofel@emlpsn01 certs]$ ll
total 8
-rw-r--r-- 1 root   root 1001 Oct 27 14:50 emlpsn01.emprd.lpemrz.com.csr
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ sudo openssl x509 -req -days 1365 -in $AMBARI_SERVER_HOSTNAME.csr -signkey $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.crt
Signature ok
subject=/C=DE/ST=Bavaria/L=Minich/O=EMNOS/CN=emlpsn01.emprd.lpemrz.com
Getting Private key
[mmartofel@emlpsn01 certs]$ ll
total 12
-rw-r--r-- 1 root   root 1192 Oct 27 14:51 emlpsn01.emprd.lpemrz.com.crt
-rw-r--r-- 1 root   root 1001 Oct 27 14:50 emlpsn01.emprd.lpemrz.com.csr
-rw-r--r-- 1 ambari root 1679 Oct 27 14:48 emlpsn01.emprd.lpemrz.com.key
[mmartofel@emlpsn01 certs]$ pwd
/etc/ambari-server/certs
[mmartofel@emlpsn01 certs]$ cd ..
[mmartofel@emlpsn01 ambari-server]$ cd conf
[mmartofel@emlpsn01 conf]$ ls -al
total 48
drwxr-xr-x 3 ambari root    248 Oct 27 14:35 .
drwxr-xr-x 7 ambari root    127 Oct 27 14:44 ..
-rwxr-xr-x 1 ambari root   8533 Oct 27 14:40 ambari.properties
-rwxr-xr-x 1 ambari root   7900 Oct 26 10:36 ambari.properties.rpmsave.20181026104227
-rwxr-xr-x 1 ambari root    317 Oct 26 18:24 krb5JAASLogin.conf
-rw-r--r-- 1 ambari ambari  317 Oct 26 18:24 krb5JAASLogin.conf.bak
-rw-r----- 1 ambari root     21 Aug 22 14:59 ldap-password.dat
-rwxr-xr-x 1 ambari root   4929 Oct 26 10:36 log4j.properties
-rwxr-xr-x 1 ambari root   2630 May 29 21:34 metrics.properties
-rw-r----- 1 ambari root     12 Jun  8 16:23 password.dat
drwxr-xr-x 2 ambari root      6 Oct 27 14:35 truststore
[mmartofel@emlpsn01 truststore]$ cd ..
[mmartofel@emlpsn01 conf]$ ls
ambari.properties  ambari.properties.rpmsave.20181026104227  krb5JAASLogin.conf  krb5JAASLogin.conf.bak  ldap-password.dat  log4j.properties  metrics.properties  password.dat  truststore
[mmartofel@emlpsn01 conf]$ pwd
/etc/ambari-server/conf
[mmartofel@emlpsn01 conf]$ sudo mv ./truststore/ /etc/ambari-server/
[mmartofel@emlpsn01 conf]$ ls
ambari.properties  ambari.properties.rpmsave.20181026104227  krb5JAASLogin.conf  krb5JAASLogin.conf.bak  ldap-password.dat  log4j.properties  metrics.properties  password.dat
[mmartofel@emlpsn01 conf]$ ll
total 48
-rwxr-xr-x 1 ambari root   8533 Oct 27 14:40 ambari.properties
-rwxr-xr-x 1 ambari root   7900 Oct 26 10:36 ambari.properties.rpmsave.20181026104227
-rwxr-xr-x 1 ambari root    317 Oct 26 18:24 krb5JAASLogin.conf
-rw-r--r-- 1 ambari ambari  317 Oct 26 18:24 krb5JAASLogin.conf.bak
-rw-r----- 1 ambari root     21 Aug 22 14:59 ldap-password.dat
-rwxr-xr-x 1 ambari root   4929 Oct 26 10:36 log4j.properties
-rwxr-xr-x 1 ambari root   2630 May 29 21:34 metrics.properties
-rw-r----- 1 ambari root     12 Jun  8 16:23 password.dat
[mmartofel@emlpsn01 conf]$ cd ..
[mmartofel@emlpsn01 ambari-server]$ ll
total 0
drwxr-xr-x 2 ambari root 117 Oct 27 14:51 certs
drwxr-xr-x 2 ambari root 230 Oct 27 14:56 conf
drwxr-xr-x 2 ambari root 131 Jun  8 15:22 conf_08_06_18_16_15.save
drwxr-xr-x 2 ambari root 127 Apr 11  2018 conf_12_04_18_10_36.save
drwxr-xr-x 2 ambari root 101 Apr 12  2018 conf_12_04_18_11_25.save
drwxr-xr-x 2 ambari root   6 Oct 27 14:35 truststore
[mmartofel@emlpsn01 truststore]$ pwd
/etc/ambari-server/truststore
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)? y
The truststore is already configured. Do you want to re-configure the truststore [y/n] (y)? y
TrustStore type [jks/jceks/pkcs12] (jks):
Path to TrustStore file :/etc/ambari-server/truststore
Password for TrustStore:
Re-enter password:
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$ pwd
/etc/ambari-server/truststore
[mmartofel@emlpsn01 truststore]$ ll
total 0
[mmartofel@emlpsn01 truststore]$ sudo keytool -import -file /etc/ambari-server/certs/emlpsn01.emprd.lpemrz.com.crt -alias ambari-server -keystore ambari-server-truststore.jks
Enter keystore password:
Re-enter new password:
Owner: CN=emlpsn01.emprd.lpemrz.com, O=EMNOS, L=Minich, ST=Bavaria, C=DE
Issuer: CN=emlpsn01.emprd.lpemrz.com, O=EMNOS, L=Minich, ST=Bavaria, C=DE
Serial number: d2977919873473e6
Valid from: Sat Oct 27 14:51:04 CEST 2018 until: Sat Jul 23 14:51:04 CEST 2022
Certificate fingerprints:
	 MD5:  CA:F2:C2:60:CF:73:81:6C:C9:B8:E6:69:B7:CB:CE:D0
	 SHA1: CA:F7:E0:B6:68:C3:C7:6B:DC:49:3A:10:3C:93:8A:28:52:B2:C2:D6
	 SHA256: B3:50:84:3A:AB:B5:84:0D:A7:8F:0F:12:BC:6D:4B:C4:51:13:E0:A6:D0:CD:F9:A5:A6:E4:72:6D:E6:FF:A8:1C
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 root root 910 Oct 27 15:03 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo chown ambari ./*
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 ambari root 910 Oct 27 15:03 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)?
The truststore is already configured. Do you want to re-configure the truststore [y/n] (y)?
TrustStore type [jks/jceks/pkcs12] (jks):
Path to TrustStore file :/etc/ambari-server/truststore/ambari-server-truststore.jks
Password for TrustStore:
Re-enter password:
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$
[mmartofel@emlpsn01 truststore]$
[mmartofel@emlpsn01 truststore]$ sudo ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 5
Do you want to configure a truststore [y/n] (y)?
Do you want to import a certificate [y/n] (y)?
Please enter an alias for the certificate: ambari-server
Enter path to certificate: /etc/ambari-server/certs/emlpsn01.emprd.lpemrz.com.crt
Ambari Server 'setup-security' completed successfully.
[mmartofel@emlpsn01 truststore]$ ll
total 4
-rw-r--r-- 1 ambari root 910 Oct 27 15:06 ambari-server-truststore.jks
[mmartofel@emlpsn01 truststore]$ sudo keytool --list --keystore ./ambari-server-truststore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


ambari-server, Oct 27, 2018, trustedCertEntry,
Certificate fingerprint (SHA1): CA:F7:E0:B6:68:C3:C7:6B:DC:49:3A:10:3C:93:8A:28:52:B2:C2:D6
[mmartofel@emlpsn01 truststore]$ sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................
Server started listening on 8080


DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
[mmartofel@emlpsn01 truststore]$ cat /etc/ambari-server/conf/ambari.properties | grep trust
kerberos.operation.verify.kdc.trust=true
ssl.trustStore.password=XXXXXXXXXXXXXXX
ssl.trustStore.path=/etc/ambari-server/truststore/ambari-server-truststore.jks
ssl.trustStore.type=jks

Re: 2.5.3 to 2.6.5 upgrade pre check fails on Ranger - Truststore path or password is not set.

jsensharma — Mon, 29 Oct 2018 18:18:03 GMT

@Marek Martofel

Wonderful!!! thank you for sharing so detailed steps.

I am marking this thread as answered.