question Re: Unable to connect remote Hadoop cluster using spark in Support Questions

Unable to connect remote Hadoop cluster using spark

skasireddy — Wed, 19 Apr 2023 17:34:39 GMT

Hi,
I'm running a reconciliation application using Spark. This application needs to connect a table from 2 different clusters Hbase table and compare the data. My application is able to connect and get the data from where the application is running on a cluster. but for remote cluster I'm getting below error
failed on local exception:

failed on local exception:exception is Failed after attempts=4, exceptions: 2023-04-19T17:11:13.491Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to Host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:14.505Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:16.522Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:19.549Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options

Re: Unable to connect remote Hadoop cluster using spark

DianaTorres — Wed, 19 Apr 2023 21:04:23 GMT

@skasireddy Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @jagadeesan and @Bharati who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Unable to connect remote Hadoop cluster using spark

jagadeesan — Wed, 19 Apr 2023 21:38:34 GMT

@skasireddy Can you please make sure you have copied hbase-site.xml from the remote HBase cluster to /etc/spark/conf/yarn-conf/ or /etc/spark/conf/ on the Edge node from where you are trying to connect your spark application?

Re: Unable to connect remote Hadoop cluster using spark

AsimShaikh — Thu, 20 Apr 2023 10:39:51 GMT

Hi @skasireddy

Can you please share spark-submit command used to trigger this application?

Also you can try specifying --files for hbase dependant files and share its outcome?

Re: Unable to connect remote Hadoop cluster using spark

skasireddy — Thu, 20 Apr 2023 18:38:53 GMT

Thanks @jagadeesan for response.
Yes, I'm creating Configuration object and setting all resources (hbase-site.xml, core-site.xml and hdfs-site.xml) for each cluster. the files I'm using are specific to Hbase cluster in configuration object

Re: Unable to connect remote Hadoop cluster using spark

skasireddy — Mon, 24 Apr 2023 14:25:26 GMT

Sorry for late response, I use oozie to submit a spark job