Reply
Explorer
Posts: 11
Registered: ‎12-01-2015

Cannot start secure DataNode Kerberos issue

Hi Expert,

I am configuring MIT kerberos for " CDH 5.4 + Isilon onefs 7.2".

Enable/config kerb on isilon work fine.

When enable/config kerb on CDH node. Restart cluster, Name node start successfully, but all DataNode start failed with below log.

 

hdfs/vIsilon-sec@FBDLSEC.LOCAL is kerb principal for Isilon cluster.

All name node/data node use the same isilon kerb principal. hdfs.

keytab is already generated and transfer to each node.

//////////////////////////////////////////////////////////////

Dec 8, 9:14:49.675 AM INFO org.apache.hadoop.hdfs.server.datanode.DataNode
registered UNIX signal handlers for [TERM, HUP, INT]
Dec 8, 9:14:50.552 AM INFO org.apache.hadoop.security.UserGroupInformation
Login successful for user hdfs/vIsilon-sec@FBDLSEC.LOCAL using keytab file /etc/hdfs.keytab
Dec 8, 9:14:50.699 AM INFO org.apache.hadoop.metrics2.impl.MetricsConfig
loaded properties from hadoop-metrics2.properties
Dec 8, 9:14:50.775 AM INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl
Scheduled snapshot period at 10 second(s).
Dec 8, 9:14:50.775 AM INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl
DataNode metrics system started
Dec 8, 9:14:50.780 AM INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner
Disabled block scanner.
Dec 8, 9:14:50.781 AM INFO org.apache.hadoop.hdfs.server.datanode.DataNode
File descriptor passing is enabled.
Dec 8, 9:14:50.783 AM INFO org.apache.hadoop.hdfs.server.datanode.DataNode
Configured hostname is clou3cdhworker5.fbdlsec.local
Dec 8, 9:14:50.789 AM FATAL org.apache.hadoop.hdfs.server.datanode.DataNode
Exception in secureMain
java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP. Using privileged resources in combination with SASL RPC data transfer protection is not supported.
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1210)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1111)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:430)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2404)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2291)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2338)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2515)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2539)
Dec 8, 9:14:50.800 AM INFO org.apache.hadoop.util.ExitUtil
Exiting with status 1

 

 

 

////////////////////////

According to CDH security guide, add below content to hdfs-site.xml

<!---General HDFS config-->
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>

<!---NameNode config-->
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/hdfs.keytab</value>
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>

<!---Secondary NameNode config-->
<property>
<name>dfs.secondary.namenode.keytab.file</name>
<value>/etc/hdfs.keytab</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.principal</name>
<value>hdfs/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>

<!---DataNode config-->
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1004</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1006</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/etc/hdfs.keytab</value>
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>hdfs/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>

<!---WEB Authentication config, if webHDFS is enabled-->
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/vIsilon-sec@FBDLSEC.LOCAL</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/etc/hdfs.keytab</value> <!-- path to the HTTP keytab -->
</property>

 

 

 

//////////////////////////////////

Add content to core-site.xml

<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>

 

 

Thanks, David

New Contributor
Posts: 1
Registered: ‎09-19-2016

Re: Cannot start secure DataNode Kerberos issue

Have the same issue with CDH 5.8 running in single user mode with kerberos.

Explorer
Posts: 24
Registered: ‎08-15-2016

Re: Cannot start secure DataNode Kerberos issue

[ Edited ]

Have you found a solution to this? I am facing the exact same issue. CDH 5.8, Kerberos with Single User Mode.

Contributor
Posts: 39
Registered: ‎02-15-2017

Re: Cannot start secure DataNode Kerberos issue

[ Edited ]

Hello David, I don't know if you can actually solved your problem.

I've to deal with the same error and maybe the solution that worked for me will be helpful for you and for other users.

Check the /etc/default/hadoop-hdfs-datanode file and uncomment the lines starting with HADOOP_SECURE_DN_USER.

This should solve the problem and you can get you DN service up and running.

 

If you have any problem please let me know.

 

 

Guido.

Announcements