Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datnanode fails after enabling Kerberos

Highlighted

Datnanode fails after enabling Kerberos

New Contributor

Hello Everybody,

I wanted to enable Kerberos on a running Server (CentOs 7.5) with Ambari 2.7.1.0
I've decided to do this with the Kerberos Wizard inside Ambari as described here: Launching the Kerberos Wizard
Therefore I installed and setup a new MIT KDC as it is described in the Documents: Install a new MIT KDC

In the final Step of the Wizard the Datanode fails to start with the following Output Error.

Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/datanode.py", line 126, in <module>

    DataNode().execute()

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 354, in execute

    self.execute_prefix_function(self.command_name, 'post', env)

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 378, in execute_prefix_function

    method(env)

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 420, in post_start

    raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file))

resource_management.core.exceptions.Fail: Pid file /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid doesn't exist after starting of the component.


When I looked in the log File of the Datanode it was empty for the given timestamp of the starting attempt.

If I disable Kerberos again all Services are starting as expected.

Does anyone have an idea or hint how I can enable Kerberos with a running Datanode afterwards?
I'm grateful for any help I'm given.

2 REPLIES 2

Re: Datnanode fails after enabling Kerberos

Super Mentor

@Philipp Schoder

After kerberizing cluster , are you able to start the DataNode Manually?

Ideally in a kerberized environment you should see a PID file with the following name "/var/run/hadoop/hdfs/hadoop_secure_dn.pid" instead of "/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid"

Example: https://community.hortonworks.com/questions/110565/what-is-hadoop-secure-dn-pid-under-varrunhadoophd...

https://community.hortonworks.com/articles/90673/why-datanodes-have-two-processes-on-a-kerberized-c....

.

Can you check if you can create an empty file with the following permission for the hdfs user?

# ls -l /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid
-rw-r--r--. 1 hdfs hadoop 6 Oct 29 06:26 /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid

.

Re: Datnanode fails after enabling Kerberos

New Contributor

@Jay Kumar SenSharma

No, if I try to start the DataNode in the Ambari Web Interface it fails with the same result.

Yes that works, I can create the empty file as hdfs user.