Unable to connect remote Hadoop cluster using spark

[#1]

Hi,
I'm running a reconciliation application using Spark. This application needs to connect a table from 2 different clusters Hbase table and compare the data. My application is able to connect and get the data from where the application is running on a cluster. but for remote cluster I'm getting below error 
failed on local exception:

failed on local exception:exception is Failed after attempts=4, exceptions: 2023-04-19T17:11:13.491Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to Host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:14.505Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:16.522Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:19.549Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options

[#2]

@skasireddy Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @jagadeesan and @Bharati  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@skasireddy Can you please make sure you have copied hbase-site.xml from the remote HBase cluster to /etc/spark/conf/yarn-conf/ or /etc/spark/conf/ on the Edge node from where you are trying to connect your spark application?

[#4]

Thanks @jagadeesan  for response.
Yes, I'm creating Configuration object and setting all resources (hbase-site.xml, core-site.xml and hdfs-site.xml) for each cluster. the files I'm using are specific to Hbase cluster in configuration object

[#5]

Hi @skasireddy 

Can you please share spark-submit command used to trigger this application?

Also you can try specifying --files for hbase dependant files and share its outcome?

[#6]

Sorry for late response, I use oozie to submit a spark job