Support Questions

daniel_kozlowsk · ‎05-17-2018

Hi All,

I have recently built HDP 2.6.4.0-91 with Ambari 2.6.1.0 on CentOS 7.4.

Installed HTTPFS and enabled SSL for it. This worked fine.

Running http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS

returned the correct output.

Now, I kerberized my cluster and followed up the steps to kerberize HTTPFS: https://community.hortonworks.com/content/supportkb/48793/configuring-httpfs-to-support-kerberos.htm...

Running:

WebBrowser: http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS
Terminal: curl -v --negotiate -X GET 'http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS'

I am getting the 404 error.

[root@ip-10-10-0-93 ~]# curl -v --negotiate -X GET 'http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS'

* About to connect() to 10.10.0.93 port 14000 (#0)
*   Trying 10.10.0.93...
* Connected to 10.10.0.93 (10.10.0.93) port 14000 (#0)
> GET /webhdfs/v1/user/sample_user/?op=LISTSTATUS HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.10.0.93:14000
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=utf-8
< Content-Length: 949
< Date: Wed, 16 May 2018 15:26:42 GMT
< 
* Connection #0 to host 10.10.0.93 left intact
<html><head><title>Apache Tomcat/6.0.48 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.48</h3></body></html>

Here is my httpfs-site.xml

<configuration>
  <!-- HUE proxy user setting -->
  <property>
    <name>httpfs.proxyuser.hue.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>httpfs.proxyuser.hue.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>httpfs.hadoop.config.dir</name>
    <value>/etc/hadoop/conf</value>
  </property>
  <property>
    <name>httpfs.authentication.type</name>
    <value>kerberos</value>
  </property>
    <property>
    <name>httpfs.hadoop.authentication.type</name>
    <value>kerberos</value>
  </property>
    <property>
    <name>httpfs.authentication.kerberos.principal</name>
    <value>HTTP/ip-10-10-0-93.ec2.internal@SUPPORT.COM</value>
  </property>
    <property>
    <name>httpfs.authentication.kerberos.keytab</name>
    <value>/etc/hadoop-httpfs/conf/httpfs-http.keytab</value>
  </property>
  <property>
    <name>httpfs.hadoop.authentication.kerberos.principal</name>
    <value>httpfs/ip-10-10-0-93.ec2.internal@SUPPORT.COM</value>
  </property>
  <property>
    <name>httpfs.hadoop.authentication.kerberos.keytab</name>
    <value>/etc/security/keytabs/httpfs-http.keytab</value>
  </property>
   <property>
      <name>httpfs.authentication.kerberos.name.rules</name>
      <value>RULE:[1:$1@$0](ambari-qa-hdp26@SUPPORT.COM)s/.*/ambari-qa/
RULE:[1:$1@$0](hdfs-hdp26@SUPPORT.COM)s/.*/hdfs/
RULE:[1:$1@$0](spark-hdp26@SUPPORT.COM)s/.*/spark/
RULE:[1:$1@$0](.*@SUPPORT.COM)s/@.*//
RULE:[2:$1@$0](beacon@SUPPORT.COM)s/.*/beacon/
RULE:[2:$1@$0](dn@SUPPORT.COM)s/.*/hdfs/
RULE:[2:$1@$0](hive@SUPPORT.COM)s/.*/hive/
RULE:[2:$1@$0](jhs@SUPPORT.COM)s/.*/mapred/
RULE:[2:$1@$0](knox@SUPPORT.COM)s/.*/knox/
RULE:[2:$1@$0](nm@SUPPORT.COM)s/.*/yarn/
RULE:[2:$1@$0](nn@SUPPORT.COM)s/.*/hdfs/
RULE:[2:$1@$0](rm@SUPPORT.COM)s/.*/yarn/
RULE:[2:$1@$0](yarn@SUPPORT.COM)s/.*/yarn/
DEFAULT</value>
    </property>
</configuration>

Any help highly appreciated!

rlevas · ‎05-17-2018

@Daniel K

I would have expected a different error; however, you need to use the FQDN (somehost.support.com) of the NN host rather than the IP address (10.10.0.93). This is so that curl can request the appropriate ticket for the service (HTTP/<FQDN>@SUPPORT.COM). But if this was your issue, you would get an HTTP 401 error back, not an HTTP 404.

The 404 error is a page/resource not found issue. I do mot believe that this is being generated by HDFS since I do not think you have properly authenticated with the server. Maybe check the Namenode logs to see if it is complaining about something.

daniel_kozlowsk · ‎06-01-2018

@Robert Levas

Thanks for coming back. Unfortunately, this does not help. All works fine in a cluster apart from HTTPFS with Kerberos enabled.

Running curl command I am getting:

[hdfs@ip-10-10-0-137 ~]$ curl http://ip-10-10-0-137.ec2.internal:14000/webhdfs/v1/user?user.name=hdfs&op=LISTSTATUS
[1] 27070
[hdfs@ip-10-10-0-137 ~]$ <html><head><title>Apache Tomcat/6.0.48 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.48</h3></body></html>

NOTE: I changed my httpfs from .93 to .137.

Any other clue?

daniel_kozlowsk · ‎06-02-2018

I have now got it working!

Review and reformat your HCC https://community.hortonworks.com/content/supportkb/48793/configuring-httpfs-to-support-kerberos.htm.... It points out httpfs-site.xml to i.e. incorrect location for keytabs.

Support Questions

HTTPFS on secure cluster not working