Datnanode fails after enabling Kerberos

[#1]

Hello Everybody,

I wanted to enable Kerberos on a running Server (CentOs 7.5) with Ambari 2.7.1.0
I've decided to do this with the Kerberos Wizard inside Ambari as described here: Launching the Kerberos Wizard
Therefore I installed and setup a new MIT KDC as it is described in the Documents: Install a new MIT KDC

In the final Step of the Wizard the Datanode fails to start with the following Output Error.

Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/datanode.py", line 126, in <module>

    DataNode().execute()

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 354, in execute

    self.execute_prefix_function(self.command_name, 'post', env)

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 378, in execute_prefix_function

    method(env)

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 420, in post_start

    raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file))

resource_management.core.exceptions.Fail: Pid file /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid doesn't exist after starting of the component.

When I looked in the log File of the Datanode it was empty for the given timestamp of the starting attempt.

If I disable Kerberos again all Services are starting as expected.

Does anyone have an idea or hint how I can enable Kerberos with a running Datanode afterwards?
I'm grateful for any help I'm given.

[#2]

@Philipp Schoder

After kerberizing cluster , are you able to start the DataNode Manually?

Ideally in a kerberized environment you should see a PID file with the following name "/var/run/hadoop/hdfs/hadoop_secure_dn.pid" instead of "/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid"

Example: https://community.hortonworks.com/questions/110565/what-is-hadoop-secure-dn-pid-under-varrunhadoophd...

https://community.hortonworks.com/articles/90673/why-datanodes-have-two-processes-on-a-kerberized-c....

.

Can you check if you can create an empty file with the following permission for the hdfs user?

# ls -l /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid
-rw-r--r--. 1 hdfs hadoop 6 Oct 29 06:26 /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid

.

[#3]

@Jay Kumar SenSharma

No, if I try to start the DataNode in the Ambari Web Interface it fails with the same result.

Yes that works, I can create the empty file as hdfs user.