Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to establish connection between 2 clusters (different HDP versions, both non-kerberized)

Highlighted

Unable to establish connection between 2 clusters (different HDP versions, both non-kerberized)

New Contributor

Hello.

We are trying to set up connection between our local HDP 2.5.6 cluster (datalab) and an external HDP 2.6.5 cluster (juggernaut). The idea is that we want to run spark (scala) jobs on yarn mode and it will call juggernaut cluster as data source.

We have set up all possible connections but even simple "hdfs dfs -ls" command from datalab to juggernaut is giving out java.io.IOException: Connection reset by peer error. I have attached debug log when I ran ls command from datalab to juggernaut.

datalab-debug-log.txt

Both datalab and juggernaut are able to ping each other, telnet ok. We have also added all juggernaut nodes to /etc/hosts on datalab. We have also observed that we are able to run "hdfs dfs -ls" command from juggernaut to datalab. In summary:

HDP 2.5.6 (datalab) --> HDP 2.6.5 (juggernaut) = NOT OK

HDP 2.6.5 (juggernaut) --> HDP 2.5.6 (datalab) = OK

Has anyone experienced something similar to this setup ang getting this issue?

2 REPLIES 2

Re: Unable to establish connection between 2 clusters (different HDP versions, both non-kerberized)

Rising Star

is both clusters, datalab and juggernaut in azure?

Re: Unable to establish connection between 2 clusters (different HDP versions, both non-kerberized)

New Contributor

hi @aengineer, azure agent is only on the juggernaut cluster (see below):

93523-azure-agent-juggernaut-cluster.jpg

is there going to be an issue with having azure storage account on juggernaut?