Gold ⭐️

Solved

Enable Kerberos - Setup KDC Account - Continue enabled but not working

20 replies
19753 views
Translate All

Hi.

I'm kerberizing an EDH 6.1.1 on SLES on Google GCE.

Kerberos is up and running.

I've created a cloudera-scm/admin@REALM and gained its ticket with kinit, so I suppose everything is fine.

I try to enable Kerberos using the CM wizard.

When I reach the Setup KDC Account I set the cloudera-scm@REALM principal and the Continue button is enabled.

I try to click on it to proceed but nothing happens.

I took a look at krb5 log files and to cloudera-scm-server log files but nothing seems to be logged.

I've doublechecked NTP, Reverse DNS and krb5.conf file but found nothing useful.

Any help ?

Thanks

Stefano

Cloudera Manager

svasiAnswer

Gold ⭐️

Hi @Shelton

Finally I've solved the problem and hope my experience could help someone else who is banging his/her head on this scenario.

At time of writing I was accessing the GCP cluster via public IP addresses.

I had a /etc/hosts file resolving hostnames to their public IP

Then customer's NOC granted me access from internal network over VPN and next attempt went fine.

Still can't understand why, maybe some networking related config issue, but being on the internal side of the network solved my problem

Simple as that!

Thanks for your precious support, and patience

Stefano

Shelton

Gold ⭐️⭐️⭐️

@svasi

Can you explain one cannot guess with the information you have given.

HDP or CDH?

OS

Steps executed

Krb5.conf

Kdc.conf

kadm5.acl

Error log or screenshot.

With the above the problem could be easier to analyze

Translate

svasiAuthor

Gold ⭐️

Thanks @Shelton

Is a CDH 6.1.1 on SUSE Enterprise 12.4 running on Google GCP compute engines.

I've installed, with zypper, those packages on the CM server:

zypper install krb5-server krb5-client krb5 openldap2-client

This is the /etc/krb5.conf file

[libdefaults]
    # "dns_canonicalize_hostname" and "rdns" are better set to false for improved security.
    # If set to true, the canonicalization mechanism performed by Kerberos client may
    # allow service impersonification, the consequence is similar to conducting TLS certificate
    # verification without checking host name.
    # If left unspecified, the two parameters will have default value true, which is less secure.
    dns_canonicalize_hostname = false
    rdns = false
    default_realm = EDHDEV.COM
    default_ccache_name = FILE:/tmp/krb5cc_%{uid}

[realms]
        EDHDEV.COM = {
                kdc = master-1.dev.edl.gcp.domain.it
                admin_server = master-1.dev.edl.gcp.domain.it
        }

[logging]
    kdc = FILE:/var/log/krb5/krb5kdc.log
    admin_server = FILE:/var/log/krb5/kadmind.log
    default = SYSLOG:NOTICE:DAEMON

This is the kdc.conf file

[kdcdefaults]
        kdc_ports = 88
        kdc_tcp_ports = 88

[realms]
        EDHDEV.COM = {
                database_name = /var/lib/kerberos/krb5kdc/principal
                admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
                acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
                dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
                key_stash_file = /var/lib/kerberos/krb5kdc/.k5.EDHDEV.COM
                kdc_ports = 88
                max_life = 1d
                max_renewable_life = 7d
        }

[logging]
    kdc = FILE:/var/log/krb5/krb5kdc.log
    admin_server = FILE:/var/log/krb5/kadmind.log

This is the kadm5.acl file:

###############################################################################
#Kerberos_principal      permissions     [target_principal]      [restrictions]
###############################################################################
#
*/admin@EDHDEV.COM  *

I've created, using kadmin.local, the cloudera-scm/admin@EDHDEV.COM

master-1:~ # kadmin.local
Authenticating as principal cloudera-scm/admin@EDHDEV.COM with password.
kadmin.local:  listprincs
K/M@EDHDEV.COM
cloudera-scm/admin@EDHDEV.COM
kadmin/admin@EDHDEV.COM
kadmin/changepw@EDHDEV.COM
kadmin/master-1.dev.edl.gcp.domain.it@EDHDEV.COM
krbtgt/EDHDEV.COM@EDHDEV.COM

I can authenticate against kerberos with kinit and gain the ticket (It takes arount 8 seconds to get the ticket):

master-1:~ # kinit cloudera-scm/admin@EDHDEV.COM
Password for cloudera-scm/admin@EDHDEV.COM:
master-1:~ # klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: cloudera-scm/admin@EDHDEV.COM

Valid starting     Expires            Service principal
11/06/19 17:58:08  11/07/19 17:58:08  krbtgt/EDHDEV.COM@EDHDEV.COM
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

I run the wizard in CM to enable Kerberos on my cluster.

I set KDC host and Encryption type according to previous output

Here I configure the principal but nothing happens when I click on Continue:

Kerberos log files:

==> /var/log/krb5/kadmind.log <==
Nov 06 12:37:51 master-1 kadmind[1083](info): set up 6 sockets
Nov 06 12:37:51 master-1 kadmind[1083](info): set up 6 sockets
Nov 06 12:37:51 master-1 kadmind[1109](info): Seeding random number generator
Nov 06 12:37:51 master-1 kadmind[1109](info): Seeding random number generator
Nov 06 12:37:51 master-1 kadmind[1109](info): starting
Nov 06 12:37:51 master-1 kadmind[1109](info): starting
Nov 06 15:18:58 master-1 kadmin.local[21785](info): No dictionary file specified, continuing without one.
Nov 06 15:18:58 master-1 kadmin.local[21785](info): No dictionary file specified, continuing without one.
Nov 06 15:20:01 master-1 kadmin.local[21922](info): No dictionary file specified, continuing without one.
Nov 06 15:20:01 master-1 kadmin.local[21922](info): No dictionary file specified, continuing without one.

==> /var/log/krb5/krb5kdc.log <==
krb5kdc: setsockopt(11,IPV6_V6ONLY,1) worked
krb5kdc: setsockopt(11,IPV6_V6ONLY,1) worked
Nov 06 17:45:15 master-1 krb5kdc[8642](info): listening on fd 12: tcp 0.0.0.0.88
Nov 06 17:45:15 master-1 krb5kdc[8642](info): listening on fd 12: tcp 0.0.0.0.88
Nov 06 17:45:15 master-1 krb5kdc[8642](info): listening on fd 11: tcp ::.88
Nov 06 17:45:15 master-1 krb5kdc[8642](info): listening on fd 11: tcp ::.88
Nov 06 17:45:15 master-1 krb5kdc[8642](info): set up 4 sockets
Nov 06 17:45:15 master-1 krb5kdc[8642](info): set up 4 sockets
Nov 06 17:45:15 master-1 krb5kdc[8643](info): commencing operation
Nov 06 17:45:15 master-1 krb5kdc[8643](info): commencing operation

cloudera-scm-server.log when I change from manage krb5.conf page to Setup KDC Account one and, then, select Continue button

2019-11-06 17:49:40,620 INFO scm-web-154:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v15/batch
2019-11-06 17:49:40,625 INFO scm-web-154:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v15/batch, Status:200
2019-11-06 17:49:40,885 INFO scm-web-380:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v15/batch
2019-11-06 17:49:40,889 INFO scm-web-380:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v15/batch, Status:200

Translate

Shelton

Gold ⭐️⭐️⭐️

@svasi

I saw a couple of potential issues and I would like you to make a few changes.

Recreate an admin for Kerberos

Using the below syntax

# kadmin.local -q "addprinc admin/admin"

----------start of output---------
Authenticating as principal root/admin@EDHDEV.COM with a password.
WARNING: no policy specified for admin/admin@EDHDEV.COM; defaulting to no policy
Enter the password for principal "admin/admin@EDHDEV.COM": welcome1
Re-enter password for principal "admin/admin@EDHDEV.COM": welcome1
Principal "admin/admin@EDHDEV.COM" created.
----------end of output---------

Validate the principal works

# kadmin -p admin/admin@EDHDEV.COM

It will ask for the password once successfully authenticated then type q to quit

Can you leave the default encryption types? and you should select Manage krb5.conf through Cloudera Manager

Don't change the kadm5.acl file it's okay the way it is !!

I saw something missing in your krb5.conf between [realms] and [logging] the domain_realm add that to all the hosts if it's a multinode

[realms]
EDHDEV.COM = {
kdc = master-1.dev.edl.gcp.domain.it
admin_server = master-1.dev.edl.gcp.domain.it
}

[domain_realm]
.edhdev.com = EDHDEV.COM
edhdev.com = EDHDEV.COM

[logging]
kdc = FILE:/var/log/krb5/krb5kdc.log
admin_server = FILE:/var/log/krb5/kadmind.log
default = SYSLOG:NOTICE:DAEMON

Go to the CM and launch the kerberization using this below the principal

admin/admin@EDHDEV.COM

Please make the above changes restart the kdc and retry kerberization it should go smoothly

Happy hadooping

Translate

Shelton

Gold ⭐️⭐️⭐️

@svasi

Can you share the files mentioned in my previous post? Redact site specific sensitive information and I will have a look at it.

Translate

svasiAuthor

Gold ⭐️

krb5.conf

[libdefaults]
    # "dns_canonicalize_hostname" and "rdns" are better set to false for improved security.
    # If set to true, the canonicalization mechanism performed by Kerberos client may
    # allow service impersonification, the consequence is similar to conducting TLS certificate
    # verification without checking host name.
    # If left unspecified, the two parameters will have default value true, which is less secure.
    dns_canonicalize_hostname = false
    rdns = false
    default_realm = EDHDEV.COM
    default_ccache_name = FILE:/tmp/krb5cc_%{uid}

[realms]
        EDHDEV.COM = {
                kdc = master-1.dev.edl.gcp.domain.it
                admin_server = master-1.dev.edl.gcp.domain.it
        }


[domain_realm]
        .edhdev.com = EDHDEV.COM
        edhdev.com = EDHDEV.COM

[logging]
    kdc = FILE:/var/log/krb5/krb5kdc.log
    admin_server = FILE:/var/log/krb5/kadmind.log
    default = SYSLOG:NOTICE:DAEMON

kdc.conf

[kdcdefaults]
        kdc_ports = 88
        kdc_tcp_ports = 88

[realms]
        EDHDEV.COM = {
                database_name = /var/lib/kerberos/krb5kdc/principal
                admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
                acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
                dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
                key_stash_file = /var/lib/kerberos/krb5kdc/.k5.EDHDEV.COM
                kdc_ports = 88
                max_life = 1d
                max_renewable_life = 7d
        }

[logging]
    kdc = FILE:/var/log/krb5/krb5kdc.log
    admin_server = FILE:/var/log/krb5/kadmind.log

kadm5.acl

###############################################################################
#Kerberos_principal      permissions     [target_principal]      [restrictions]
###############################################################################
#
*/admin@EDHDEV.COM  *

Translate

svasiAuthor

Gold ⭐️

Just tried to set up a different cluster, also on Google Cloud Platform, with a CDH 5.15 to try a new setup on another version.

Same configuration, same behaviour.

Another setup local to my datacenter has no problem.

Does anybody met same problem on GCP?

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.