Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do I reference a high availability cluster in WEBHDFS?

How do I reference a high availability cluster in WEBHDFS?

New Contributor

My question is how to use WEBHDFS in a High Availability cluster. (It is really a broader question than WEBHDFS, since any REST API that has to reference the active name node has the same issue.)

(The following examples are based on a cluster with Name Service ID: HDPDRV1; Name nodes XXXXn1 (active), XXXXn2 (standby); Data nodes XXXXd1 – XXXXd10)

If I issue the following curl command against the active name node, the OPEN is successful and I retrieve the requested file:

curl -vL 
"http://XXXXn1:50070/webhdfs/v1/user/fred/msg/messages.txt?op=OPEN&user.name=fred"

However, if I issue the same curl command against the standby name node:

curl -vL 
"http://XXXXn2:50070/webhdfs/v1/user/fred/msg/messages.txt?op=OPEN&user.name=fred"

I get an error: “Operation category READ is not supported in state standby”

I have read the HDP 2.3/2.4 literature and it suggests that I should use the NameService ID, rather than the name nodes addresses.

However, If I try to use the name service ID (HDPDRV1)

curl -vL  http://HDPPRD1:50070/webhdfs/v1/user/fred/msg/messages.txt?op=OPEN&user.name=fred

I get the error: “Couldn't resolve host 'HDPDRV1'” (which is perfectly reasonable, since the name service ID is not defined in DNS)

Prior to HDP2.3, I would use a VIP (Virtual IP) in this case and the VIP mechanism would sort out the active name node.

I have read that HDP clients should reference a local copy of the HDP config files to sort out what name nodes are associated with the name service Id. However, in the WEBHDFS environment there is no client.

The (very) surprising thing is that a WEBHDFS CREATE command works, whether it points to the active or standby name node. I do not understand why a CREATE can reference the standby name node while an OPEN cannot.

e.g. The following command, referencing the standby name node (XXXXn2), is successful (as is the equivalent command referencing the active name node):

curl -vL -T messages.txt -X PUT http://XXXXn2:50070/webhdfs/v1/user/fred/msg/messages.txt?op=CREATE&overwrite=true&user.name=fred

Even though XXXXn2 is the STANDBY name node, it successfully redirects this request to a data node and the CREATE succeeds:

* About to connect() to XXXXn2 port 50070 (#0)

* Trying 39.7.48.3... connected

* Connected to XXXXn2 (39.7.48.3) port 50070 (#0)

> PUT /webhdfs/v1/user/fred/msg/messages.txt?op=CREATE&overwrite=true&user.name=fred HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10

> Host: XXXXn2:50070

> Accept: */*

> Content-Length: 34370

> Expect: 100-continue

>

< HTTP/1.1 100 Continue

< HTTP/1.1 307 TEMPORARY_REDIRECT

< Cache-Control: no-cache

< Expires: Fri, 14 Oct 2016 00:07:19 GMT

< Date: Fri, 14 Oct 2016 00:07:19 GMT

< Pragma: no-cache

< Expires: Fri, 14 Oct 2016 00:07:19 GMT

< Date: Fri, 14 Oct 2016 00:07:19 GMT

< Pragma: no-cache

< Set-Cookie: hadoop.auth="u=fred&p=fred&t=simple&e=1476439639126&s=1HtrrYJVY6T4lvSVR/EKZcCDzEI="; Path=/; HttpOnly

<

* Connection #0 to host XXXXn2 left intact

* Issue another request to this URL: 'http://XXXXd4:50075/webhdfs/v1/user/fred/msg/messages.txt?op=CREATE&user.name=fred&namenoderpcaddress=HDPDRV1&createflag=&createparent=true&overwrite=true'

* About to connect() to XXXXd4 port 50075 (#1)

* Trying 39.7.48.15... connected

* Connected to XXXXd4 (39.7.48.15) port 50075 (#1)

> PUT /webhdfs/v1/user/fred/msg/messages.txt?op=CREATE&user.name=fred&namenoderpcaddress=HDPDRV1&createflag=&createparent=true&overwrite=true HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10

> Host: XXXXd4:50075

> Accept: */*

> Content-Length: 34370

> Expect: 100-continue

>

< HTTP/1.1 100 Continue

< HTTP/1.1 201 Created

< Location: hdfs://HDPDRV1/user/fred/msg/messages.txt

< Content-Length: 0

< Connection: close

< Location: http://XXXXd4:50075/webhdfs/v1/user/fred/msg/messages.txt?op=CREATE&user.name=fred&namenoderpcaddres...

< Content-Type: application/octet-stream

< Content-Length: 0

< Server: Jetty(6.1.26.hwx)

Is it correct that a CREATE can use either name node but an OPEN must use the active name node?

What is the preferred way of addressing the active name node in REST API calls?

Thanks

Tony

2 REPLIES 2

Re: How do I reference a high availability cluster in WEBHDFS?

Nameservice ID should be mentioned without ports.

Knox can help you achieve this in an alternate way. Check this doc :-

https://community.hortonworks.com/articles/49668/knox-using-webhdfs-with-namenode-ha.html

Highlighted

Re: How do I reference a high availability cluster in WEBHDFS?

New Contributor

Hi Tony,

we experienced similar issues and we do agree that namenodes should redirect all operations to the active namenode. There is a long standing JIRA open on this issue (although mentioning write operations), but unfortunately no visible activity on this one: HDFS-6371 https://issues.apache.org/jira/browse/HDFS-6371

Another workaround for this issue is to use HttpFS https://hadoop.apache.org/docs/r2.7.1/hadoop-hdfs-httpfs/

Be careful though - HttpFS can easily become a bottleneck in scenarios where high throughput is required, as it proxies not only the "commands", but also the "payload".

Best regards

Michael

Don't have an account?
Coming from Hortonworks? Activate your account here