Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Connection between clusters with and without Kerberos

Solved Go to solution
Highlighted

Connection between clusters with and without Kerberos

Cloudera Employee

Hi there! how are you? =)

We have a HDP 2.2 cluster (without Kerberos) already on Production, and we are planning to build another HPD 2.4 cluster (with Kerberos) on the side of the first one.

So, cutting to the chase, my doubt is: Are we going to have some troubles on the attempting to connect to a kerberos enabled cluster?

Are there limitations on the communication between clusters? I mean, will a cluster be able to connect to the cluster for read/write operations, without getting kerberos errors.

As far as I know, this is not a supported configuration but I am looking for someone that has experience configuring this solution.

Thanks in advance.

Thiago.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Connection between clusters with and without Kerberos

@Thiago

If you are using DistCp and WebHDFS to copy data between a secure cluster and an nonsecure cluster by doing the following:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-c8ffaa14-ea...

  1. Set ipc.client.fallback-to-simple-auth-allowed to true in core-site.xml on the secure cluster side:
    <property> 
      <name>ipc.client.fallback-to-simple-auth-allowed</name>
      <value>true</value> 
    </property>
  2. Use below commands from the secure cluster side.
    distcp webhdfs://nonsecure-cluster webhdfs://secure-cluster 
    distcp webhdfs://secure-cluster webhdfs://nonsecure-cluster 
    
2 REPLIES 2

Re: Connection between clusters with and without Kerberos

@Thiago

If you are using DistCp and WebHDFS to copy data between a secure cluster and an nonsecure cluster by doing the following:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-c8ffaa14-ea...

  1. Set ipc.client.fallback-to-simple-auth-allowed to true in core-site.xml on the secure cluster side:
    <property> 
      <name>ipc.client.fallback-to-simple-auth-allowed</name>
      <value>true</value> 
    </property>
  2. Use below commands from the secure cluster side.
    distcp webhdfs://nonsecure-cluster webhdfs://secure-cluster 
    distcp webhdfs://secure-cluster webhdfs://nonsecure-cluster 
    

Re: Connection between clusters with and without Kerberos

Hello @Thiago. It is possible to achieve communication across secured and unsecured clusters. A common use case for this is using DistCp for transfer of data between clusters.

As mentioned in other answers, the configuration property ipc.client.fallback-to-simple-auth-allowed=true tells a secured client that it may enter a fallback unsecured mode when the unsecured server side fails to satisfy authentication. However, I recommend not setting this in core-site.xml, and instead setting it on the command line invocation specifically for the DistCp command that needs to communicate with the unsecured cluster. Setting it in core-site.xml means that all RPC connections for any application are eligible for fallback to simple authentication. This potentially expands the attack surface for man-in-the-middle attacks.

Here is an example of overriding the setting on the command line while running DistCp:

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo

The command must be run while logged into the secured cluster, not the unsecured cluster.

This is adapted from one of my prior answers:

https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.ht...