Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HTTPFS on secure cluster not working

Highlighted

HTTPFS on secure cluster not working

New Contributor

Hi All,

I have recently built HDP 2.6.4.0-91 with Ambari 2.6.1.0 on CentOS 7.4.

Installed HTTPFS and enabled SSL for it. This worked fine.

Running http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS

returned the correct output.

Now, I kerberized my cluster and followed up the steps to kerberize HTTPFS: https://community.hortonworks.com/content/supportkb/48793/configuring-httpfs-to-support-kerberos.htm...

Running:

I am getting the 404 error.

[root@ip-10-10-0-93 ~]# curl -v --negotiate -X GET 'http://10.10.0.93:14000/webhdfs/v1/user/sample_user/?op=LISTSTATUS'

* About to connect() to 10.10.0.93 port 14000 (#0)
*   Trying 10.10.0.93...
* Connected to 10.10.0.93 (10.10.0.93) port 14000 (#0)
> GET /webhdfs/v1/user/sample_user/?op=LISTSTATUS HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.10.0.93:14000
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=utf-8
< Content-Length: 949
< Date: Wed, 16 May 2018 15:26:42 GMT
< 
* Connection #0 to host 10.10.0.93 left intact
<html><head><title>Apache Tomcat/6.0.48 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.48</h3></body></html>

Here is my httpfs-site.xml

<configuration>
  <!-- HUE proxy user setting -->
  <property>
    <name>httpfs.proxyuser.hue.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>httpfs.proxyuser.hue.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>httpfs.hadoop.config.dir</name>
    <value>/etc/hadoop/conf</value>
  </property>
  <property>
    <name>httpfs.authentication.type</name>
    <value>kerberos</value>
  </property>
    <property>
    <name>httpfs.hadoop.authentication.type</name>
    <value>kerberos</value>
  </property>
    <property>
    <name>httpfs.authentication.kerberos.principal</name>
    <value>HTTP/ip-10-10-0-93.ec2.internal@SUPPORT.COM</value>
  </property>
    <property>
    <name>httpfs.authentication.kerberos.keytab</name>
    <value>/etc/hadoop-httpfs/conf/httpfs-http.keytab</value>
  </property>
  <property>
    <name>httpfs.hadoop.authentication.kerberos.principal</name>
    <value>httpfs/ip-10-10-0-93.ec2.internal@SUPPORT.COM</value>
  </property>
  <property>
    <name>httpfs.hadoop.authentication.kerberos.keytab</name>
    <value>/etc/security/keytabs/httpfs-http.keytab</value>
  </property>
   <property>
      <name>httpfs.authentication.kerberos.name.rules</name>
      <value>RULE:[1:$1@$0](ambari-qa-hdp26@SUPPORT.COM)s/.*/ambari-qa/
RULE:[1:$1@$0](hdfs-hdp26@SUPPORT.COM)s/.*/hdfs/
RULE:[1:$1@$0](spark-hdp26@SUPPORT.COM)s/.*/spark/
RULE:[1:$1@$0](.*@SUPPORT.COM)s/@.*//
RULE:[2:$1@$0](beacon@SUPPORT.COM)s/.*/beacon/
RULE:[2:$1@$0](dn@SUPPORT.COM)s/.*/hdfs/
RULE:[2:$1@$0](hive@SUPPORT.COM)s/.*/hive/
RULE:[2:$1@$0](jhs@SUPPORT.COM)s/.*/mapred/
RULE:[2:$1@$0](knox@SUPPORT.COM)s/.*/knox/
RULE:[2:$1@$0](nm@SUPPORT.COM)s/.*/yarn/
RULE:[2:$1@$0](nn@SUPPORT.COM)s/.*/hdfs/
RULE:[2:$1@$0](rm@SUPPORT.COM)s/.*/yarn/
RULE:[2:$1@$0](yarn@SUPPORT.COM)s/.*/yarn/
DEFAULT</value>
    </property>
</configuration>

Any help highly appreciated!

3 REPLIES 3

Re: HTTPFS on secure cluster not working

@Daniel K

I would have expected a different error; however, you need to use the FQDN (somehost.support.com) of the NN host rather than the IP address (10.10.0.93). This is so that curl can request the appropriate ticket for the service (HTTP/<FQDN>@SUPPORT.COM). But if this was your issue, you would get an HTTP 401 error back, not an HTTP 404.

The 404 error is a page/resource not found issue. I do mot believe that this is being generated by HDFS since I do not think you have properly authenticated with the server. Maybe check the Namenode logs to see if it is complaining about something.

Re: HTTPFS on secure cluster not working

New Contributor

@Robert Levas

Thanks for coming back. Unfortunately, this does not help. All works fine in a cluster apart from HTTPFS with Kerberos enabled.

Running curl command I am getting:

[hdfs@ip-10-10-0-137 ~]$ curl http://ip-10-10-0-137.ec2.internal:14000/webhdfs/v1/user?user.name=hdfs&op=LISTSTATUS
[1] 27070
[hdfs@ip-10-10-0-137 ~]$ <html><head><title>Apache Tomcat/6.0.48 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.48</h3></body></html>

NOTE: I changed my httpfs from .93 to .137.

Any other clue?

Re: HTTPFS on secure cluster not working

New Contributor

I have now got it working!

Review and reformat your HCC https://community.hortonworks.com/content/supportkb/48793/configuring-httpfs-to-support-kerberos.htm.... It points out httpfs-site.xml to i.e. incorrect location for keytabs.