Member since
09-29-2015
362
Posts
242
Kudos Received
63
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2159 | 03-14-2019 01:00 PM | |
| 2468 | 01-23-2019 04:19 PM | |
| 9269 | 01-15-2019 01:59 PM | |
| 7250 | 01-15-2019 01:57 PM |
01-09-2019
03:34 PM
@huzaira bashir
It seems like there may be an issue exporting keytab files from the KDC. Can you try to do this manually using the admin user you configured in Ambari? /bin/kinit -c /tmp/my_cc -S kadmin/<KDC Admin Server Host> <KDC Admin Principal Name>
/bin/kadmin -c /tmp/my_cc -s <KDC Admin Server Host> -r <Realm> -q "xst -k /tmp/ambari_tmp.keytab -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-qa-<Cluster Name>@<Realm>"
/bin/klist -kte /tmp/ambari_tmp.keytab You will need to change the specifics to match your cluster. For my cluster I am using:
KDC Admin Server Host: c7402.ambari.apache.org KDC Admin Principal Name: admin/admin!@EXAMPLE.COM Realm: EXAMPLE.COM Cluster Name: c1 Also, I assume that you haven't changed:
The default temporary directory: /tmp The kadmin principal name: kadmin/<KDC Admin Server Host> The format of the Ambari smoke user principal name: ambari-qa-<Cluster Name>@<Realm> The default encryption types: aes des3-cbc-sha1 rc4 des-cbc-md5 Using the commands from above, I get the following: [root@c7402 ~]# /bin/kinit -c /tmp/my_cc -S kadmin/c7402.ambari.apache.org admin/admin@EXAMPLE.COM
Password for admin/admin@EXAMPLE.COM:<br>[root@c7402 ~]# /bin/kadmin -c /tmp/my_cc -s c7402.ambari.apache.org -r EXAMPLE.COM -q "xst -k "/tmp/ambari_tmp.keytab" -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-server-c1@EXAMPLE.COM"
Authenticating as principal admin/admin@EXAMPLE.COM with existing credentials.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Administration credentials NOT DESTROYED.<br>[root@c7402 ~]# /bin/klist -kte /tmp/ambari_tmp.keytab
Keytab name: FILE:/tmp/ambari_tmp.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des3-cbc-sha1)
4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (arcfour-hmac)
4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des-cbc-md5)
4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes256-cts-hmac-sha1-96) Can you try this to see if you get any errors?
... View more
01-04-2019
02:35 PM
@huzaira bashir Is the Ambari server on a host that is registered with the cluster? If not, I was recently alerted to an issue where this case was causing an error. But enabling Kerberos would have failed for you... unless you had enabled Kerberos before upgrading to Ambari 2.7.1. See AMBARI-25088 - Enable Kerberos fails when Ambari server is not on a registered host.
... View more
01-03-2019
02:57 PM
@huzaira bashir Can you take a look at your Ambari server log (/var/log/ambari-server/ambari-server.log) and see if there are any interesting error messages?
... View more
11-15-2018
02:16 PM
I am not exactly sure all of the tasks that the Ambari agent reset operation performs. It seems like it will do a lot more than just clean up the existing certs - if it does that at all. However, it will not perform any of the Ambari server-side tasks. So at lest the Ambari server steps from the article need to take place manually.
... View more
08-27-2018
04:01 PM
@Anant
Aneja
You probably should have posed this as a question in the form, rather than a comment to this article. It may have gotten answered quicker. The rule you are using will not perform the translation you want. The regular expression syntax to match using case-insensitivity is not supported as you have specified it and the translation will not generate local names with all lower-case characters. The rule you want is more like RULE:[1:$1@$0](.*@FOO.COM)s////L With this rule, the Hadoop UGI class will translate user@FOO.COM to user@foo.com [root@c7401 ~]# hadoop org.apache.hadoop.security.HadoopKerberosName joe_user@FOO.COM
18/08/27 15:57:07 INFO util.KerberosName: Non-simple name joe_user@FOO.COM after auth_to_local rule RULE:[1:$1@$0](.*@FOO.COM)s////L
Name: joe_user@FOO.COM to joe_user@foo.com As for the other principal names, they will technically be invalid since the realm name needs to always be in all upper-case characters. user1@FOO.COM - legal user2@foo.com - illegal user3@Foo.com - illegal
... View more
08-08-2018
01:38 PM
I am glad you found this useful. Thanks for the note.
... View more
12-22-2017
03:20 PM
I just tried this and had no issues. Ambari 2.4.2/HDP 2.4 curl command worked fine Express upgrade to HDP 2.5 (Ambari 2.4.2/HDP 2.5) curl command worked fine Check your ambari.log file to see if there are any interesting errors.
... View more
12-21-2017
03:26 PM
1 Kudo
Hi @Mahesh Thumar Given the same version of Ambari (version 2.2), I am not sure why the stack would make a difference. On that note, as far as I know, nothing has changed for that entry point since Ambari 2.1.0. When I get a chance I will see if I can figure out what the deal is.
... View more
06-12-2017
03:29 PM
1 Kudo
There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections. In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents. On the Ambari server:
Stop the Ambari server Backup /var/lib/ambari-server/keys and it child directories Delete the following files from /var/lib/ambari-server/keys ca.key ca.csr ca.crt pass.txt keystore.p12 *.csr *.crt Delete the following files from /var/lib/ambari-server/keys/db index.txt.old index.txt.attr.old serial.old Truncate the following files from /var/lib/ambari-server/keys/db index.txt index.txt.attr Edit the following files from /var/lib/ambari-server/keys/db serial set the contents to be exactly 00
Delete all files under /var/lib/ambari-server/keys/db/newcerts Restart Ambari server. On each Ambari agent host: Stop the Ambari agent Backup /var/lib/ambari-agent/keys and it child directories Delete the following files from /var/lib/ambari-agent/keys ca.crt *.crt *.csr *.key Restart Ambari agent After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file: 12 Jun 2017 14:38:19,606 INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096 was finished with exit code: 0 - the operation was completely successfully
.
12 Jun 2017 14:38:19,640 INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
peration was completely successfully.
12 Jun 2017 14:38:19,683 INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,701 INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.
....
12 Jun 2017 14:52:53,797 INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org
12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
12 Jun 2017 14:52:53,849 INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.
After restarting the Ambari agent, the following (or similar) entires should be seen in the /var/log/ambari-agent/ambari-agent.log file: INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading
INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request
INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr"
INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request
INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.
... View more
Labels:
06-12-2017
03:03 PM
There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections. In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents. On the Ambari server:
Stop the Ambari server Backup /var/lib/ambari-server/keys and it child directories Delete the following files from /var/lib/ambari-server/keys ca.key ca.csr ca.crt pass.txt keystore.p12 *.csr *.crt Delete the following files from /var/lib/ambari-server/keys/db index.txt.old index.txt.attr.old serial.old Truncate the following files from /var/lib/ambari-server/keys/db index.txt index.txt.attr Edit the following files from /var/lib/ambari-server/keys/db serial set the contents to be exactly 00
Delete all files under /var/lib/ambari-server/keys/db/newcerts Restart Ambari server. On each Ambari agent host: Stop the Ambari agent Backup /var/lib/ambari-agent/keys and it child directories Delete the following files from /var/lib/ambari-agent/keys ca.crt *.crt *.csr *.key Restart Ambari agent After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file: 12 Jun 2017 14:38:19,606 INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096 was finished with exit code: 0 - the operation was completely successfully
.
12 Jun 2017 14:38:19,640 INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
peration was completely successfully.
12 Jun 2017 14:38:19,683 INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,701 INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.
....
12 Jun 2017 14:52:53,797 INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org
12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
12 Jun 2017 14:52:53,849 INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.
After restarting the Ambari agent, the following (or similar) entires should be seen in the /var/log/ambari-agent/ambari-agent.log file: INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading
INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request
INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr"
INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request
INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.
... View more
Labels: