Member since
01-19-2017
3679
Posts
632
Kudos Received
372
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 814 | 06-04-2025 11:36 PM | |
| 1396 | 03-23-2025 05:23 AM | |
| 692 | 03-17-2025 10:18 AM | |
| 2499 | 03-05-2025 01:34 PM | |
| 1627 | 03-03-2025 01:09 PM |
10-15-2017
07:03 AM
@ilia kheifets Sorry to hear you are encountering all these problems. Could you tell me the HDP,Ambari and OS type and version you are trying to install. I will try to guide you.
... View more
10-13-2017
07:16 PM
@tsharma Can you attach the following files in here - krb5.conf, - kdc.conf and - kadm5.acl. What is your AD Domain? Did you import the AD cert? self signed or CA ?
... View more
10-13-2017
04:50 PM
@tsharma Can you have a look and this HCC doc and get back to me. I think that's what you intend to implement. Create MIT KDC in the Hadoop cluster to manage service principals while using a one-way trust to allow AD users to utilze the Hadoop environment seamlessly
... View more
10-13-2017
01:59 PM
@tsharma You have a couple of things wrong in your Kerberos setup. I noticed in your cluster the REALM is MIT.EDU, it should be different and NOT as you stated MIT.EDU (Michigan Institue of Technology) creators of Kerberos.Your kadm5.acl is wrong ! Your REALM is not ATHENA.MIT.EDU, that an example given with the MIT documentation. How did the container "KnoxUsers" and "knxadmin" user get created in AD? Below is the procedure on a Centos/RHEL but the commands are similar on all UNIX/LINUX OS'es Assumptions: REALM is TEST.COM Install the KDC server. The below command will deliver the nessary configuration files # yum install krb5-server Edit your /etc/krb5.conf replace all occurrences of TEST and test please match case(lower or upper) # cat /etc/krb5.conf The krb5.conf should look like this please notice the entries in lowercase for test.com and .test.com [libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = TEST.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = /tmp/krb5cc_%{uid}
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
test.com = TEST.COM
.test.com = TEST.COM
[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log
[realms] TEST.COM =
{ admin_server = {your_kdc_server}
kdc = {your_kdc_server} }
Modify the kdc.conf in /var/kerberos/krb5kdc/kdc.conf replace the TEST.COM with your REALM cat /var/kerberos/krb5kdc/kdc.conf The kdc.conf should look like this [kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms] TEST.COM =
{ #master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal Delete the existing KDC database (optional) # kdb5_util destroy -f TEST.COM (delete Kerberos existing TEST database) The below will prompt you for the password please note them and don't lose it. Create the KDC database is didn't already # kdb5_util create -s TEST.COM (create Kerberos database) When the above process terminates you will have created a database for the TEST.COM (realm) Start the kadmin & KDC # service krb5kdc start
# service kadmin start To autostart the KDC and Kadmin on system bootup execute the below step # chkconfig krb5kdc on
# chkconfig kadmin on The value for the kadm5.acl should match the output of the below command. On the KDC as root run # kdestroy (ONLY if you already created a previous REAL database) Create principal # kadmin.local -q "addprinc admin/admin"
Authenticating as principal root/admin@TEST with password.
WARNING: no policy specified for admin/admin@TEST;
defaulting to no policy Enter password for principal "admin/admin@TEST":
Re-enter password for principal "admin/admin@TEST":
Principal "admin/admin@TEST" created. Validate you can connect to the KDC with admin rights # kadmin.local
Authenticating as principal root/admin@TEST.COM with password. From the above your kadm5.acl in /var/kerberos/krb5kdc/kadm5.acl should be,notice the space between the.COM and * */admin@TEST.COM * Validate that get a valid kerberos ticket # kinit admin/admin@TEST
Password for admin/admin@TEST Check validity it should be 7 days according to your krb5.conf # klist
Ticket cache: FILE:/tmp/krb5cc_0 Default
principal: admin/admin@TEST
Valid starting Expires Service principal
10/13/2017 15:48:43 10/14/2017 15:48:43 krbtgt/TEST.COM@TEST.COM Now you can trigger the Ambari -Kerberos wizard and walk through the steps, the input values you need are Admin principal and Admin principal password The keytabs should now be generated successfully, in your case single node check on the VM in /etc/security/keytabs Please let me know if that helped
... View more
10-12-2017
11:08 PM
@Neha G Here you go !
... View more
10-12-2017
10:42 PM
@Neha G To get your Hadoop realm from Ambari UI -->Admin-->Kerberos -->General under the Global settings should be your REALM The AD REALM could be domainController.example.com A Kerberos realm is not a Windows 2000 domain,
... View more
10-12-2017
09:46 PM
@D G Would you be able to find the task attempt that actually failed. That task attempt can show you which machine and YARN container is ran on. Sometimes the logs dont have the error because it logged into stderr. In that case, the stderr from the containers YARN logs may show the error. Could you set this variable and re-run the query set hive.execution.engine=tez;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask=true;
set hive.auto.convert.join.noconditionaltask.size=405306368;
set hive.vectorized.execution.enabled=true;
set hive.vectorized.execution.reduce.enabled =true;
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
set hive.merge.mapfiles =true;
set hive.merge.mapredfiles=true;
set hive.merge.size.per.task=134217728;
set hive.merge.smallfiles.avgsize=44739242;
set mapreduce.job.reduce.slowstart.completedmaps=0.8 Please let me know if that helped
... View more
10-12-2017
09:14 PM
@Sai Sandeep, See the location in the attached screenshot! You can navigate to those directories for the hive view ,file view errors 🙂
... View more
10-12-2017
05:01 PM
@Sai Sandeep, Can you paste the error stack in hive20-view.log located in /var/log/ambari-server/hive20-view
... View more
10-12-2017
08:49 AM
@Neha G Your krb5.conf on all hosts should have both entries od the HDP and AD realms see below. In the below example HDP.HORTONWORKS.COM = HDP domain AD.HORTONWORKS.COM= AD domain [realms]
HDP.HORTONWORKS.COM ={
kdc = kdc-server.hortonworks.com
admin_server = kdc-server.hortonworks.com
default_domain = hdp.hortonworks.com
}
AD.HORTONWORKS.COM ={
kdc = ad-server.hortonworks.com
admin_server = ad-server.hortonworks.com
default_domain = ad.hortonworks.com
} Hope that helps
... View more