Reply
New Contributor
Posts: 4
Registered: ‎10-18-2016

Cloudera Director fails to bootstrap cluster with Kerberos in AWS

Hi mighty all,

 

I am trying to bootstrap Kerebrezied Cluder cluster using Cloudera director in AWS and miserable fail that. Looking for word of advise.Here is my setup:

 

I have setup two differnet directory services in AWS - simple AD (mycompany.local) and Microsoft AD ( mycompany.dev) in the same VPC with Cloudera cluster just for the sake of simplicity of my expiriment. I use Centos 7.2 with latest updates for all nodes.

 

I have configured Director bootstrap conf:

 

 

krbAdminUsername: "Administrator@mydomain.local"
# The password for the administrative Kerberos account.
krbAdminPassword: hidden

cloudera-manager {
configs {
CLOUDERA_MANAGER {
enable_api_debug: true
# The type of KDC Cloudera Manager will be using. Valid values are "MIT KDC" and "Active Directory"
KDC_TYPE: "Active Directoriy"
# The KDC host name or IP address.
KDC_HOST: "pdc1.mydomain.local"
# The security realm that your KDC uses. This will be of the format of a fully qualified domain name: YOUR.KDC.REALM
SECURITY_REALM: "cloudera.mydomain.local"
# The Active Directory KDC domain. Only applicable to Active Directory KDCs. This will be in the format of an X.500 Directory Specification:
# DC=domain,DC=example,DC=com
AD_KDC_DOMAIN: "DC=mydomain,DC=local"
# Allow Cloudera Manager to deploy Kerberos configurations to hosts. This should be set to true unless you have an alternate mechanism to generate or retrieve the
# Kerberos configuration on your Cloudera Manager node instances.
KRB_MANAGE_KRB5_CONF: true
# The encryption types your KDC supports. Some of those listed below will require the unlimited strength JCE policy files.
KRB_ENC_TYPES: "aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc"
}
}

 

Without kerberos settings cluster comes up clean.  With the settings above I got these errors:

 

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: com.cloudera.launchpad.bootstrap.ApiCommandFailedException: Import of Kerberos admin principal credentials failed: /usr/share/cmf/bin/import_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf3704429855939634101.keytab
+ USER=Administrator@mydomain.local
+ PASSWD='REDACTED'
+ KVNO=1
+ SLEEP=0
+ RHEL_FILE=/etc/redhat-release
+ '[' -f /etc/redhat-release ']'
+ set +e
+ grep Tikanga /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'CentOS release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'Scientific Linux release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ set -e
+ '[' -z /var/run/cloudera-scm-server/krb53386771840768236265.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb53386771840768236265.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb53386771840768236265.conf
+ IFS=' '
+ read -a ENC_ARR
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e aes256-cts'
+ ktutil
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e aes128-cts'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e des3-hmac-sha1'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e arcfour-hmac'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e des-hmac-sha1'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e des-cbc-md5'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p Administrator@mydomain.local -k 1 -e des-cbc-crc'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ echo 'wkt /var/run/cloudera-scm-server/cmf3704429855939634101.keytab'
+ chmod 600 /var/run/cloudera-scm-server/cmf3704429855939634101.keytab
+ kinit -k -t /var/run/cloudera-scm-server/cmf3704429855939634101.keytab Administrator@mydomain.local
kinit: Cannot find KDC for realm "dfsi.local" while getting initial credentials

How I can configure Cloudera Director to boostrap kerbersied  cluster in AWS with AD/SimpleAD in AWS? I have run out of ideas how to make it work and need help of experts here.

 

 

Cloudera Employee
Posts: 3
Registered: ‎10-17-2016

Re: Cloudera Director fails to bootstrap cluster with Kerberos in AWS

[ Edited ]

Can you try the following?

1- Terminate clusters

2- Delete the generated principals in AD from the previous installations

3- Remove the following property (or set it to the default of false):

 

KRB_MANAGE_KRB5_CONF: false

 4- Relaunch the cluster. 

 

Setting KRB_MANAGE_KRB5_CONF to true requires some other settings which you may not need.

New Contributor
Posts: 4
Registered: ‎10-18-2016

Re: Cloudera Director fails to bootstrap cluster with Kerberos in AWS

Thanks Fahds, I will certanly try that. Small correction - I do not have generated principals in AD as Cloudera couldn't connect/find AD. This is newly genreated AWS AD  service just for that installation.

 

What other settings would be reqiuered to use that?  

KRB_MANAGE_KRB5_CONF: true

I have just folllowed kerebors bootrap guide/exmple and all ofthem use this option... 

Anyway I will try to rebuild a cluster and let you know how did it go...

Highlighted
New Contributor
Posts: 4
Registered: ‎01-04-2016

Re: Cloudera Director fails to bootstrap cluster with Kerberos in AWS

Spotted a typo in the KDC_TYPE entry, hopefully this might help resolving the issue of Cloudera not finding AD.

 

 

# The type of KDC Cloudera Manager will be using. Valid values are "MIT KDC" and "Active Directory"
KDC_TYPE: "Active Directoriy"

 

New Contributor
Posts: 4
Registered: ‎10-18-2016

Re: Cloudera Director fails to bootstrap cluster with Kerberos in AWS


Edmond wrote:

Spotted a typo in the KDC_TYPE entry, hopefully this might help resolving the issue of Cloudera not finding AD.

 

 

# The type of KDC Cloudera Manager will be using. Valid values are "MIT KDC" and "Active Directory"
KDC_TYPE: "Active Directoriy"

 


Ok, I have found the problem. Apparently Cloudera doesn't support AWS AD/Simply AD.

 

I have build my own KDC server on Linux. Changed settings of KDC_TYPE: to MIT and was able to  configure cludera cluster from a manager. should be able to bootstrap with those settings from director too. Cloudera documentation doesn't cover that limitation.... Has anyone faced that problem before?

 

Although after server was confiugured with KDC succesfully hue service stoppped working... Not usre why but it would be different topic.

 

Thanks Edmond! Really appciate you help. 

Cloudera Employee
Posts: 3
Registered: ‎10-17-2016

Re: Cloudera Director fails to bootstrap cluster with Kerberos in AWS

[ Edited ]

Update to my previous reply: KRB_MANAGE_KRB5_CONF should always be set to true. After testing it, currently director does not support this setting to be false.

Announcements