HDFS RPC

[#1]

Hi,

I am trying to set up RPC HDFS remote access, but can find nothing online that explains the steps required.

Can anyone provide any links/resources etc to assist?

Evan

[#2]

Hello @getschwifty ,

thank you for enquiring about how to set up RPC HDFS remote access. Please see this blog article about WebHDFS that should do the job.

Please let us know if we addressed your enquiry by pressing the "Accept as Solution" button. It also helps search engines to surface the answers that were actually helpful.

Best regards:

Ferenc

[#3]

Hi,

Webhdfs is apparently not going to work. The stakeholders are trying to use mlflow and pyarrow. We're currently dealing with AD realm issues that is delaying progress.

I was able to test communicating between separate clusters with:

hdfs dfs -ls hdfs://core15.fqdn:8022/

Those clusters are signed by the same certificate authority however (and share the same AD realm). The new hosts currently are possibly not - I have not been able to get the stakeholders to finalise that.

With RPC calls, what does the remote host require? Same domain? Does it actually require the same signed certificates? Does it need the destination cluster core-site and hdfs-site files?

Currently, the errors I am being sent are (which I think are related to the realm issues):

(base) bash-4.2$ hdfs dfs -ls hdfs://core15.fqdn:8022/

20/07/20 17:26:23 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host1073.fqdn/xx.x.xxxx.xx"; destination host is: "core15.fqdn":8022;

[#4]

Hello @getschwifty ,

did you have a chance to test if cross-realm trust works OK between your nodes from different realms?

Kind regards:

Ferenc