@kwabstian53  Yes, I did as you said.   telnet gspidn01.gsp.local 636 Able to connect,   telnet gspidn01.gsp.local 636 Trying 10.32.83.35... Connected to gspidn01.gsp.local. Escape character is '^]'.   As you said I did change ldap.conf, #TLS_CACERTDIR /home/admin/ad-ca.crt TLS_CACERT /etc/ssl/certs/ca-bundle.crt TLS_REQCERT allow   Did restart ambari-server.   Even after this also I am unable to do ldapsearch with 636 port.   ... View more

Hi @kwabstian53 ,   Thank you for the quick response. here are the result for below tests, [root@gspdhd01 admin]# openssl s_client -connect gspidn01.gsp.local:636 CONNECTED(00000003) depth=1 DC = local, DC = gsp, CN = gsp-GSPIDN01-CA verify return:1 depth=0 CN = gspidn01.gsp.local verify return:1 --- Certificate chain 0 s:/CN=gspidn01.gsp.local i:/DC=local/DC=gsp/CN=gsp-GSPIDN01-CA --- Server certificate -----BEGIN CERTIFICATE----- MIIF4TCCBMmgAwIBAgITGAAAAAObh58/1Hp3NQAAAAAAAzANBgkqhkiG9w0BAQUF ADBGMRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxEzARBgoJkiaJk/IsZAEZFgNnc3Ax GDAWBgNVBAMTD2dzcC1HU1BJRE4wMS1DQTAeFw0xOTA1MTYxMzI2NDdaFw0yMDA1 MTUxMzI2NDdaMB0xGzAZBgNVBAMTEmdzcGlkbjAxLmdzcC5sb2NhbDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAL0NAv82q4G18yxxAqkIBcN6HF6xfn8o hPyAy0NEg6oN9DYPMZeAbI+M+4PgSFCkahhHq+Cc1hk920wuSkayCbLhGIrbQxk5 t66nYAccquRoUrcZEilIh3dlSFn7jUV5uNd6J4BJWeds7ZTbUWcPUv6LyaqHCYAH zifCQJc72VEZcyrfYHVKCRHFNP/wbc0dmIhsBPlrE8MfCpZmRCGk6dWMnTeQJxjG WEK03GuUohSPAyvRUszvws5ss8nclK0aNc3so3d4ChdHu3ES8LcI/EKX4Q+HZvFm gsIbP+1n82aY7w1ytI3Rr/q2FEfPszWsRFHN0prpUXk6UYDcCWexBXUCAwEAAaOC Au8wggLrMC8GCSsGAQQBgjcUAgQiHiAARABvAG0AYQBpAG4AQwBvAG4AdAByAG8A bABsAGUAcjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDgYDVR0PAQH/ BAQDAgWgMHgGCSqGSIb3DQEJDwRrMGkwDgYIKoZIhvcNAwICAgCAMA4GCCqGSIb3 DQMEAgIAgDALBglghkgBZQMEASowCwYJYIZIAWUDBAEtMAsGCWCGSAFlAwQBAjAL BglghkgBZQMEAQUwBwYFKw4DAgcwCgYIKoZIhvcNAwcwHQYDVR0OBBYEFJBJUps0 1o3qXXNBLI5eB6IZkv/MMB8GA1UdIwQYMBaAFIF0wvRzUzMxfPE1XswM76Z9nrWK MIHMBgNVHR8EgcQwgcEwgb6ggbuggbiGgbVsZGFwOi8vL0NOPWdzcC1HU1BJRE4w MS1DQSxDTj1nc3BpZG4wMSxDTj1DRFAsQ049UHVibGljJTIwS2V5JTIwU2Vydmlj ZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1nc3AsREM9bG9jYWw/ Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERp c3RyaWJ1dGlvblBvaW50MIG/BggrBgEFBQcBAQSBsjCBrzCBrAYIKwYBBQUHMAKG gZ9sZGFwOi8vL0NOPWdzcC1HU1BJRE4wMS1DQSxDTj1BSUEsQ049UHVibGljJTIw S2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1n c3AsREM9bG9jYWw/Y0FDZXJ0aWZpY2F0ZT9iYXNlP29iamVjdENsYXNzPWNlcnRp ZmljYXRpb25BdXRob3JpdHkwPgYDVR0RBDcwNaAfBgkrBgEEAYI3GQGgEgQQSctt 2ckXq0y5qQCiaAaA34ISZ3NwaWRuMDEuZ3NwLmxvY2FsMA0GCSqGSIb3DQEBBQUA A4IBAQAAW36YLTpHiiRjSWmu6H0/SjCbeLmdKJN5s1XnbXt4kjbbCUYvTMbb/oJ/ h5uf7kIsRdl0zfncGD/JsepLeVLh3GKz1ZDhOWkHQW4VbX0KUW84yqv+irxuKosd KDuhvGpaR2D9KmlYTdfzDF53rzvyBm6hZUQW+au9E/5MQ3Ej8XnjgaEK5GL3UKNE S3uUhqtdK91PcirvpTRVdgGsJb3DkzvxC628d3VQKLKkio4YkXi9rE3/rongu85C ow5WZ4SaPFh63l93Kd+Raa7CNmn1IWA0HXCAmX5kjNrQW9LDtYjTnvcXXfrnwaXd HNApJDvKPHlbqc6UGBU7JoUj6ri8 -----END CERTIFICATE----- subject=/CN=gspidn01.gsp.local issuer=/DC=local/DC=gsp/CN=gsp-GSPIDN01-CA --- No client certificate CA names sent Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 --- SSL handshake has read 1726 bytes and written 659 bytes --- New, TLSv1/SSLv3, Cipher is AES128-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : AES128-SHA256 Session-ID: 4220000056B7D77DBE11C53E29A71B0E71E06A867EF394A5564B9AE70D546C48 Session-ID-ctx: Master-Key: C2E4A6977EF6CF2B62C396EBF9C49E0DA95035CAA5A08BEC0C23A406F95DBA0C4B16EECC89F5CAEE504A00C597D7ED25 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1567053717 Timeout : 300 (sec) Verify return code: 0 (ok) [root@gspdhd01 admin]# openssl s_client -connect gspidn01.gsp.local:636 -CAfile /home/admin/ad-ca.crt CONNECTED(00000003) depth=1 DC = local, DC = gsp, CN = gsp-GSPIDN01-CA verify return:1 depth=0 CN = gspidn01.gsp.local verify return:1 --- Certificate chain 0 s:/CN=gspidn01.gsp.local i:/DC=local/DC=gsp/CN=gsp-GSPIDN01-CA --- Server certificate -----BEGIN CERTIFICATE----- MIIF4TCCBMmgAwIBAgITGAAAAAObh58/1Hp3NQAAAAAAAzANBgkqhkiG9w0BAQUF ADBGMRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxEzARBgoJkiaJk/IsZAEZFgNnc3Ax GDAWBgNVBAMTD2dzcC1HU1BJRE4wMS1DQTAeFw0xOTA1MTYxMzI2NDdaFw0yMDA1 MTUxMzI2NDdaMB0xGzAZBgNVBAMTEmdzcGlkbjAxLmdzcC5sb2NhbDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAL0NAv82q4G18yxxAqkIBcN6HF6xfn8o hPyAy0NEg6oN9DYPMZeAbI+M+4PgSFCkahhHq+Cc1hk920wuSkayCbLhGIrbQxk5 t66nYAccquRoUrcZEilIh3dlSFn7jUV5uNd6J4BJWeds7ZTbUWcPUv6LyaqHCYAH zifCQJc72VEZcyrfYHVKCRHFNP/wbc0dmIhsBPlrE8MfCpZmRCGk6dWMnTeQJxjG WEK03GuUohSPAyvRUszvws5ss8nclK0aNc3so3d4ChdHu3ES8LcI/EKX4Q+HZvFm gsIbP+1n82aY7w1ytI3Rr/q2FEfPszWsRFHN0prpUXk6UYDcCWexBXUCAwEAAaOC Au8wggLrMC8GCSsGAQQBgjcUAgQiHiAARABvAG0AYQBpAG4AQwBvAG4AdAByAG8A bABsAGUAcjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDgYDVR0PAQH/ BAQDAgWgMHgGCSqGSIb3DQEJDwRrMGkwDgYIKoZIhvcNAwICAgCAMA4GCCqGSIb3 DQMEAgIAgDALBglghkgBZQMEASowCwYJYIZIAWUDBAEtMAsGCWCGSAFlAwQBAjAL BglghkgBZQMEAQUwBwYFKw4DAgcwCgYIKoZIhvcNAwcwHQYDVR0OBBYEFJBJUps0 1o3qXXNBLI5eB6IZkv/MMB8GA1UdIwQYMBaAFIF0wvRzUzMxfPE1XswM76Z9nrWK MIHMBgNVHR8EgcQwgcEwgb6ggbuggbiGgbVsZGFwOi8vL0NOPWdzcC1HU1BJRE4w MS1DQSxDTj1nc3BpZG4wMSxDTj1DRFAsQ049UHVibGljJTIwS2V5JTIwU2Vydmlj ZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1nc3AsREM9bG9jYWw/ Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERp c3RyaWJ1dGlvblBvaW50MIG/BggrBgEFBQcBAQSBsjCBrzCBrAYIKwYBBQUHMAKG gZ9sZGFwOi8vL0NOPWdzcC1HU1BJRE4wMS1DQSxDTj1BSUEsQ049UHVibGljJTIw S2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1n c3AsREM9bG9jYWw/Y0FDZXJ0aWZpY2F0ZT9iYXNlP29iamVjdENsYXNzPWNlcnRp ZmljYXRpb25BdXRob3JpdHkwPgYDVR0RBDcwNaAfBgkrBgEEAYI3GQGgEgQQSctt 2ckXq0y5qQCiaAaA34ISZ3NwaWRuMDEuZ3NwLmxvY2FsMA0GCSqGSIb3DQEBBQUA A4IBAQAAW36YLTpHiiRjSWmu6H0/SjCbeLmdKJN5s1XnbXt4kjbbCUYvTMbb/oJ/ h5uf7kIsRdl0zfncGD/JsepLeVLh3GKz1ZDhOWkHQW4VbX0KUW84yqv+irxuKosd KDuhvGpaR2D9KmlYTdfzDF53rzvyBm6hZUQW+au9E/5MQ3Ej8XnjgaEK5GL3UKNE S3uUhqtdK91PcirvpTRVdgGsJb3DkzvxC628d3VQKLKkio4YkXi9rE3/rongu85C ow5WZ4SaPFh63l93Kd+Raa7CNmn1IWA0HXCAmX5kjNrQW9LDtYjTnvcXXfrnwaXd HNApJDvKPHlbqc6UGBU7JoUj6ri8 -----END CERTIFICATE----- subject=/CN=gspidn01.gsp.local issuer=/DC=local/DC=gsp/CN=gsp-GSPIDN01-CA --- No client certificate CA names sent Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 --- SSL handshake has read 1726 bytes and written 659 bytes --- New, TLSv1/SSLv3, Cipher is AES128-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : AES128-SHA256 Session-ID: 1D230000F115D38D94169A6DBA55FE2062091ADD61B826DCF01BBC3AA9289224 Session-ID-ctx: Master-Key: 934F0B8AEF0695606BEF9BF0112F889E7E0499D15DED3744963307CC84BC2C4058E903136ABF1CC4274D820C4063D571 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1567053874 Timeout : 300 (sec) Verify return code: 0 (ok) [root@gspdhd01 admin]# ldapsearch -h gspidn02.gsp.local -p 636 -x -D "uid=HDP_Service,ou=ServiceAccounts,dc=gsp,dc=local" -b "dc=gsp,dc=local" -W Enter LDAP Password: ldap_result: Can't contact LDAP server (-1) [root@gspdhd01 admin]# ldapsearch -h gspidn01.gsp.local -p 636 -x -D "uid=HDP_Service,ou=ServiceAccounts,dc=gsp,dc=local" -b "dc=gsp,dc=local" -W Enter LDAP Password: ldap_result: Can't contact LDAP server (-1)   May I know where is the problem?   Regards, Manjunath P N ... View more

Hi,   This gave some hope that we are in proper direction, [root@10 security]# curl "ldaps://10.32.83.35:636/DC=gsp.local?cn,objectClass?sub?(objectClass=)" -u "cn=HDP_Service,OU=Service Accounts,DC=gsp,DC=local" --cacert /etc/pki/ca-trust/source/anchors/activedirectory.pem -v Enter host password for user 'cn=HDP_Service,OU=Service Accounts,DC=gsp,DC=local': * About to connect() to 10.32.83.35 port 636 (#0) * Trying 10.32.83.35... * Connected to 10.32.83.35 (10.32.83.35) port 636 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * Closing connection 0 curl: (77) Problem with the SSL CA cert (path? access rights?) How can I change .pem file path?  ... View more

Here in this step one of the team members did follow some different steps and installed a certificate but now we are unable to remove that whenever I key tool its saying key tool already present <ambari-server> can it be removed completely so that I can start the fresh installation? If yes could please let me know the steps to completely remove the existing certificate. ... View more

curl "ldaps://10.32.83.35:636/DC=gsp.local?cn,objectClass?sub?(objectClass=)" -u "cn=HDP_Service,OU=Service Accounts,DC=gsp,DC=local" --insecure -v Enter host password for user 'cn=HDP_Service,OU=Service Accounts,DC=gsp,DC=local': * About to connect() to 10.32.83.35 port 636 (#0) * Trying 10.32.83.35... * Connected to 10.32.83.35 (10.32.83.35) port 636 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * NSS: client certificate not found (nickname not specified) * SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA256 * Server certificate: * subject: CN=gspidn01.gsp.local * start date: May 16 13:26:47 2019 GMT * expire date: May 15 13:26:47 2020 GMT * common name: gspidn01.gsp.local * issuer: CN=gsp-GSPIDN01-CA,DC=gsp,DC=local * LDAP local: ldaps://10.32.83.35:636/DC=gsp.local?cn,objectClass?sub?(objectClass=) * LDAP remote: search failed Success 0000202B: RefErr: DSID-0310082F, data 0, 1 access points ref 1: 'gsp.local' * Closing connection 0 curl: (39) NSS: client certificate not found (nickname not specified)   Output for the 1st command shared.   Not able to access the AD certificate using OpenSSL.   Regards, ... View more

Hi,   The document is really handy but for me, the issue has been resolved after moving the Ambari metrics to different node initially it was in data node now moved to Namenode and I am still finding the reason how it works here.   Regards, Manjunath P N ... View more

Hi ngarg,   The document is really handy but for me the issue has been resolved after moving the ambari metrics to different node initially it was in datanode now moved to Namenode and I am still finding the reason how it works here.   Regards, Manjunath P N ... View more

Thank you Jsen, I would also like to know once after the installation of the certificate will I be able to use curl command example: curl ldaps://exmaple.com:636 because currently i get error like  curl ldaps://example.com:636 curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate. But it works well with ldap://example.com:389. This means the installation of SSL is not done in the proper way right.   Regards, Manjunath ... View more

@Geoffrey Shelton Okot Please find the link for new thread https://community.hortonworks.com/questions/246319/failed-to-connect-to-kdc-failed-to-communicate-wit.html Please guide me on this its really critical for me. ... View more

Hi @Robert Levas I have confirmed the SSL path password type and even I did set. kerberos.operation.verify.kdc.trust = false but still I am receiving an error like Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://hostip:636: hostip:636 Update the KDC settings in krb5-conf and kerberos-env configurations to correct this issue. Please guide me on this Regards, Manjunath P N ... View more

Hi @Geoffrey Shelton Okot I have a quick question while configuring Kerberos with AD in Ambari wizard we have some prerequisites one of them is AD's SSL, is it really mandatory? If yes can we add the certificate later once after enabling Kerberos? Regards, Manjunath P N ... View more

Hi @Hamid Zorgani, Thanks a lot for the detailed information. It's really helpful. I have couple of doubts could you please clarify, 1. Do we need to execute all steps in Ambari server? especially from step 1 to 10? 2. I have OpenLDAP installed already in my in cluster, even are also all steps mandatory? Please let me know if you need any more information. Regards, Manjunath P N ... View more

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 73, in <module> HdfsClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 35, in install import params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params.py", line 25, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py", line 343, in <module> dn_principal_name = dn_principal_name.replace('_HOST',hostname.lower()) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'dfs.datanode.kerberos.principal' was not found in configurations dictionary ... View more

@Rishi, @Gaurav Sharma @Jay Kumar SenSharma Please guide me on the similar issue. resource_management.core.exceptions.ExecutionFailed: Execution of 'kinit -kt /etc/security/keytabs/hdfs.headless.keytab ' returned 127. -bash: kinit: command not found raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'dfs.datanode.kerberos.principal' was not found in configurations dictionary! I am getting these error when i restart Hadoop services. I disabled Kerberos and removed and manually removed these files : rm -rf /var/kerberos/ rm /etc/krb5.conf rm -rf /usr/lib64/krb5 Now i dont know whats causing this issue please guide me on this. Regards, Manjunath P N ... View more

same issue with me please suggest some solution. @benoit moisan did you find any solution??? Please guide me. ... View more

Hi Robert, Could you please clarify me some doubts. I have installed kerberos in my cluster and its working fine. Now i have to enable HA for Kerberos so as per my understanding I should install KDC in another server which acts as Standby and then I should update krb5.conf file on both servers as mentioned above. Is my understanding correct? if not could you please guide me through the steps to enable HA. Kind Regards, Manjunath P N ... View more

@Geoffrey Shelton Okot @Jay Kumar SenSharma When I am trying to enable the Kerberos after all the back end setup, i am getting warning as "YARN log and local dir will be deleted and ResourceManager state will be formatted as part of Enabling/Disabling Kerberos. " what does it mean local dir and what all will be deleted and how does it related because YARN log will be deletion is acceptable but why local dir? Can you please provide some detailed clarification on this? ... View more

@Jay Kumar SenSharma, Thank you for your inputs. After running yum install commands the outcome will be krb5.conf file and i am already having the same file structure in my repository so can I directly go ahead with enabling Kerberos via Ambari and does the conf file will accept this? Regards, Manjunath P N ... View more

Hi @Jay Kumar SenSharma, Thank you for your inputs. When I am trying to install Kerberos using "yum install krb5-server krb5-libs krb5-workstation" it is failing to install with below error messages. https://fedora-mirror.zerocopy.io/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to fedora-mirror.zerocopy.io:443; Connection refused" Trying other mirror. http://mirror.de.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2a00:c98:2030:a034::21: Network is unreachable" Trying other mirror. http://ftp.uni-stuttgart.de/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2001:7c0:2041:8::112: Network is unreachable" Trying other mirror. http://mirror.23media.de/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2a00:f48:1007::80: Network is unreachable" Trying other mirror. https://mirror.imt-systems.com/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2a01:7e0:0:201::10:20: Network is unreachable" Trying other mirror. ^Chttp://fedora.tu-chemnitz.de/pub/linux/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] Error importing repomd.xml for epel: Damaged repomd.xml file Trying other mirror. Error list keeps on increasing what could be the reason for this? Regards, Manjunath P N ... View more

Hi @mthiele, One quick question: Does ambari server and all other datanodes will have krb5.conf file by default? or it will be available under /etc folder only after we enabling kerberos via ambari? Because when I see in our prod env. krb5.file is available even though we did not enable. If yes, after Configuring KDC in Ambari does it change the conf for all nodes? Regards, Manjunath P N ... View more

@Geoffrey Shelton Okot, Could you please share the link for screenshot which you have attached. Regards, Manjunath P N ... View more

Same question from my side. ... View more

Hi @Geoffrey Shelton Okot, I too have the same question but if we are upgrading Spark 2.4 manually does it supports HDP with all other dependencies? Because as you have mentioned it has its own spark with version 2.3.2. Regards, Manjunath P N ... View more

Thank you very much for the detailed suggestions. I will keep you updated on the results, this is really useful. ... View more