Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

webHDFS 403 error

Highlighted

webHDFS 403 error

New Contributor

Hello all,

after fresh kerberization of Ambari 2.7.3 / HDP 3 cluster, the HDFS namenode isn't able to start because the hdfs user can't talk to the webhdfs. The following error is returned:

GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)

It is not only from ambari: I can recreate this error from a simple curl call from hdfs user:

su - hdfs
curl --negotiate -u : http://datanode:50070/webhdfs/v1/tmp?op=GETFILESTATUS

Which returns

</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/tmp. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</pre></p>
</body>
</html>

Overall permission for this user should be in tact, since I'm able to run hdfs operations from shell and kinit without problems. What could be the problem?

I've tried recreating keytabs several times, and fiddling with ACL settings on the config, but nothing works. What principal is WEBHDFS expecting? The same results are when I'm trying accessing it with HTTP/host@EXAMPLE.COM principal.

NB: I'll add that there's nothing fancy in the HDFS settings, mainly stock/default config.

NB2: I will add, that I've added all possible encryption types to krb5.conf as I could find, but none if these helped:

    default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
    default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
    permitted_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

@Geoffrey Shelton Okot

6 REPLIES 6

Re: webHDFS 403 error

Mentor

@Javert Kirilov

I just run the command against a kerberized 2.6.5 and all is okay

$ curl --negotiate -u : http://panda:50070/webhdfs/v1/tmp?op=GETFILESTATUS
{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":7,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1539249388688,"owner":"hdfs","pathSuffix":"","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}

Let me spin up an HDP 3.x matching your spec then kerberize it and will revert. are you using Active Directory Kerberos authentication


Re: webHDFS 403 error

Mentor

@Javert Kirilov

I have just completed the kerberization of a Single node Centos 7 HDP3.1.0.0 running Ambari 2.7.3.0 everything went as planned no hiccups.

See attached screenshots

krb5.conf

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = UGANDA.UG
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  .uganda.ug = UGANDA.UG
  uganda.ug = UGANDA.UG
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  UGANDA.UG = {
    admin_server = pawor.uganda.ug
    kdc = pawor.uganda.ug
  }

kadm5.acl

*/admin@UGANDA.UG       *

kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 UGANDA.UG = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

To help in the diagnostics can you align your configuration to mine mapping your REALM and Server values and retest! Can you try start the services manually, run the start date nodes command on all the data nodes

su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start secondarynamenode"
su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-datanode/../hadoop/sbin/hadoop-daemon.sh start datanode"

Take a keen interest in the supported_enctypes and compare with your cluster I don't think adding all the encryption types will help:-)


kirilov02.pngkirilov04.pngkirilov03.pngkirilov01.png

Re: webHDFS 403 error

Mentor

@Javert Kirilov

Resend the linkedin invitation deleted by error

Re: webHDFS 403 error

New Contributor
@Geoffrey Shelton Okot

Thanks, I think I solved it.

You know what was the problem? The Ambari wasn't creating/re-creating keytabs and principals for HTTP/_HOST@DOMAIN.COM - had to do that by hand. Plus, with the correct encryption...

Thank you for your help!

It's just interesting: did you have to create HTTP/_HOST principal, or did the Ambari create it automatically for you? If that's the case, I wonder why it didn't on my machine. By the way, I'm using openLDAP for Ldap/Kerberos database.

Re: webHDFS 403 error

Mentor

@Javert Kirilov

Glad to know your case was resolved.......No everything was generated automatically by running the Kerberos Wizard.

Re: webHDFS 403 error

New Contributor

@Geoffrey Goldman

Important question (should I post it as a new question? It does kind of follow up from your latest comment, so I post it here): so how should ideally the "default_tkt_enctypes", "default_tgs_enctypes" and "permitted_enctypes" should look like for a normal HDP cluster (not a test sandbox), which would work 100% of the times and also provide high level security?

1. When I've tried the default suggested settings of "des3-cbc-sha1 des3-hmac-sha1 des3-cbc-sha1-kd", I would get errors that the security level was too low. I've then further added "aes256-cts-hmac-sha1-96", but it seems more than one decent enctype is required for proper encryption?

2. The default Kerberos settings, suggested by Ambari, also suggests "des3-cbc-sha1 des3-hmac-sha1 des3-cbc-sha1-kd", but comments it out by default, so I guess it ends up using some default values, which doesn't seem stable (what if the default will change over time or new version of kerberos).

3. Now I've added all possible configs, "aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal", but when using ``xst -k `` from ``kadmin`` service, it exports arounds 2-3 entries in the keytab with different encryptions, but not all 8+. Suggesting, that only some types are actually important.