Support Questions
Find answers, ask questions, and share your expertise

HDFS RPC

HDFS RPC

Explorer

Hi,

 

I am trying to set up RPC HDFS remote access, but can find nothing online that explains the steps required.

 

Can anyone provide any links/resources etc to assist?

 

Evan

3 REPLIES 3
Highlighted

Re: HDFS RPC

Moderator

Hello @getschwifty ,

 

thank you for enquiring about how to set up RPC HDFS remote access. Please see this blog article about WebHDFS that should do the job.

 

Please let us know if we addressed your enquiry by pressing the "Accept as Solution" button. It also helps search engines to surface the answers that were actually helpful.

 

Best regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Highlighted

Re: HDFS RPC

Explorer

Hi,

 

Webhdfs is apparently not going to work. The stakeholders are trying to use mlflow and pyarrow. We're currently dealing with AD realm issues that is delaying progress.

 

I was able to test communicating between separate clusters with:

 

hdfs dfs -ls hdfs://core15.fqdn:8022/

 

Those clusters are signed by the same certificate authority however (and share the same AD realm). The new hosts currently are possibly not - I have not been able to get the stakeholders to finalise that.

 

With RPC calls, what does the remote host require? Same domain? Does it actually require the same signed certificates? Does it need the destination cluster core-site and hdfs-site files?

 

Currently, the errors I am being sent are (which I think are related to the realm issues):

(base) bash-4.2$ hdfs dfs -ls hdfs://core15.fqdn:8022/

20/07/20 17:26:23 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host1073.fqdn/xx.x.xxxx.xx"; destination host is: "core15.fqdn":8022;

Highlighted

Re: HDFS RPC

Moderator

Hello @getschwifty ,

 

did you have a chance to test if cross-realm trust works OK between your nodes from different realms?

 

Kind regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Don't have an account?