Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

WebHDFS over Knox returns an error although retry mechanism

Highlighted

WebHDFS over Knox returns an error although retry mechanism

Rising Star

Hi,

I'm implementing a client application that performs webHdfs calls through Knox/kerberos mechanism.

Recently, it received an error :

"Unable to connect to the remote server, on URL=https://XXXXXXXX:8443/gateway/pam/webhdfs/v1/MyPath/VLG/DTS0004?op=MKDIRS

The remote server returned an error: (404) Not Found."

However, the directory has been properly created in HDFS...

I know that high availability configuration for knox ensures a retry mechanism and I guess this caused the right creation of the folder in the end, but I'm wondering why my client application received an error...

Any clue will be greatly appreciated !

Thanks

5 REPLIES 5

Re: WebHDFS over Knox returns an error although retry mechanism

Super Mentor

@Sebastien Chausson

Which version of HDP are you using?

I tried the following on HDP 2.6 Sandbox. (ambari 2.5)

Are you using "PUT" request as following?

$ curl -k -u guest:guest -X PUT https://localhost:8443/gateway/ui/webhdfs/v1/demo/test2?op=MKDIRS
{"boolean":true}

- Later i could see the directory was created properly:

$ curl -k -u guest:guest -X GET https://localhost:8443/gateway/ui/webhdfs/v1/demo?op=LISTSTATUS
{
  "FileStatuses": {
    "FileStatus": [
      {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum": 3,
        "fileId": 17501,
        "group": "hdfs",
        "length": 0,
        "modificationTime": 1492628598063,
        "owner": "hdfs",
        "pathSuffix": "data",
        "permission": "700",
        "replication": 0,
        "storagePolicy": 0,
        "type": "DIRECTORY"
      },
      {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum": 0,
        "fileId": 50150,
        "group": "hdfs",
        "length": 0,
        "modificationTime": 1495534620699,
        "owner": "admin",
        "pathSuffix": "test",
        "permission": "755",
        "replication": 0,
        "storagePolicy": 0,
        "type": "DIRECTORY"
      },
      {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum": 0,
        "fileId": 50363,
        "group": "hdfs",
        "length": 0,
        "modificationTime": 1495536742602,
        "owner": "dr.who",
        "pathSuffix": "test1",
        "permission": "755",
        "replication": 0,
        "storagePolicy": 0,
        "type": "DIRECTORY"
      },
      {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum": 0,
        "fileId": 50408,
        "group": "hdfs",
        "length": 0,
        "modificationTime": 1495537177483,
        "owner": "dr.who",
        "pathSuffix": "test2",
        "permission": "755",
        "replication": 0,
        "storagePolicy": 0,
        "type": "DIRECTORY"
      }
    ]
  }
}

- I could see that the directory was created properly and no 500 Error.

Following is my "/etc/knox/conf/topologies/ui.xml" file entry for "webhdfs" section.

<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>Anonymous</name>
            <enabled>true</enabled>
        </provider>
        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>false</enabled>
        </provider>
    </gateway>
    <service>
        <role>AMBARI</role>
        <url>http://localhost:8080</url>
    </service>
    <service>
        <role>AMBARIUI</role>
        <url>http://localhost:8080</url>
    </service>
    <service>
        <role>RANGER</role>
        <url>http://sandbox.hortonworks.com:6080/</url>
    </service>
    <service>
        <role>RANGERUI</role>
        <url>http://sandbox.hortonworks.com:6080/</url>
    </service>

    <!-- ADDED WEBHDFS -->
    <service>
        <role>WEBHDFS</role>
        <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
    </service>

</topology>

.

.

Re: WebHDFS over Knox returns an error although retry mechanism

Super Mentor

@Sebastien Chausson

If you are using NN HA then you should also refer to the WebHdfs HA section on https://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS

To enable HA functionality for WebHDFS in Knox the following configuration has to be added to the topology file:

<provider>
   <role>ha</role>
   <name>HaProvider</name>
   <enabled>true</enabled>
   <param>
       <name>WEBHDFS</name>
       <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
   </param>
</provider><br>

And for the service configuration itself the additional URLs that standby nodes should be added to the list. The active URL (at the time of configuration) should ideally be added to the top of the list.

<service>
    <role>WEBHDFS</role>
    <url>http://{host1}:50070/webhdfs</url>
    <url>http://{host2}:50070/webhdfs</url>
</service><br>

Re: WebHDFS over Knox returns an error although retry mechanism

Rising Star

In case, Knox topology is pointing to Standby Name Node in HA scenario, you will get a 403 error and logs will point to something like this "message": "Operation category WRITE is not supported in state standby". In above case, OP is getting 404 error however the directory is getting created in HDFS.

Re: WebHDFS over Knox returns an error although retry mechanism

Rising Star

@Jay

Thanks for your help.

In fact, everything is properly configured on the cluster (or seems to be), and everything usually works fine. I just noticed this problem once and I tried to figure out what was the issue (because my client application is not necessarily "robust" enough to comply with such errors. It relies on out-of-the-box HA features, but if theses features return unexpected errors, we have to handle them accordingly...Do you know how the retry mechanism is supposed to work ?

Regards

Re: WebHDFS over Knox returns an error although retry mechanism

Rising Star

Can you provide gateway.log and gateway-audit.log files from /var/log/knox directory for further debugging?