Cloudera manager 5.5 install : HDFS error

rkalz — Fri, 16 Sep 2022 09:56:47 GMT

Hi,

I'm trying to install a single node cluster with the cloudera manager installer.

Everything goes smoothly until Cluster Setup fails at the "Start HDFS" step with the error "Failed to format NameNode".

On the same screen when clicking the "Full log file" link I get a popup with :

 No such file or directory: '/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-myhost.log.out'

I checked the dfs.namenode.name.dir property, it has two values "/data/dfs/nn" and "/soft/dfs/nn", both of these directories seem to have suficient permissions:

# ll /data/dfs
drwx------ 2 hdfs hadoop 4096 Jan 12 10:57 nn

# ll /soft/dfs
drwx------ 3 hdfs hadoop 4096 Jan 11 16:14 dn
drwx------ 3 hdfs hadoop 4096 Jan 11 16:14 nn

I also checked the log directory permissions (for the popup error message), they are as follows:

# ll /var/log
drwxrwxr-x 4 hdfs   hadoop  4096 Jan 12 11:11 hadoop-hdfs

Here are logs I got in cloudera-scm-agent.log when hitting the Retry button :

[12/Jan/2016 11:26:08 +0000] 62644 MainThread agent        INFO     Deleting process 19-hdfs-NAMENODE-format
[12/Jan/2016 11:26:08 +0000] 62644 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Retiring process 19-hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread util         INFO     Using specific audit plugin for process hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread util         INFO     Creating metadata plugin for process hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread util         INFO     Using specific metadata plugin for process hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread util         INFO     Using generic metadata plugin for process hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        WARNING  Not creating a monitor for 20-hdfs-NAMENODE-format: MonitorDaemon not created yet
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     First time to activate the process [20-hdfs-NAMENODE-format].
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Created /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Chowning /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format to hdfs (453) hdfs (171)
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format to 0751
[12/Jan/2016 11:26:09 +0000] 62644 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.5.1-1.cdh5.5.1.p0.11'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[12/Jan/2016 11:26:09 +0000] 62644 MainThread parcel       INFO     The following requested parcels are not available: {}
[12/Jan/2016 11:26:09 +0000] 62644 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[12/Jan/2016 11:26:09 +0000] 62644 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.5.1-1.cdh5.5.1.p0.11'}
[12/Jan/2016 11:26:09 +0000] 62644 MainThread util         INFO     Extracted 20 files and 0 dirs to /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format.
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Created /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format/logs
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Chowning /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format/logs to hdfs (453) hdfs (171)
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format/logs to 0751
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     reading limits: {u'limit_memlock': None, u'limit_fds': None}
[12/Jan/2016 11:26:09 +0000] 62644 MainThread agent        INFO     Triggering supervisord update.
[12/Jan/2016 11:26:09 +0000] 62644 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hdfs-NAMENODE-format with pipelines [NavigatorLog4jAppender,HdfsAuditLogger]
[12/Jan/2016 11:26:09 +0000] 62644 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hdfs-NAMENODE-format with pipelines []
[12/Jan/2016 11:26:09 +0000] 62644 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hdfs-NAMENODE-format with pipelines [NavigatorLog4jAppender,HdfsAuditLogger]
[12/Jan/2016 11:26:09 +0000] 62644 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline 'NavigatorLog4jAppender' for hdfs-NAMENODE-format (log dir: None)
[12/Jan/2016 11:26:09 +0000] 62644 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline 'HdfsAuditLogger' for hdfs-NAMENODE-format (log dir: None)
[12/Jan/2016 11:26:09 +0000] 62644 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hdfs-NAMENODE-format with pipelines []
[12/Jan/2016 11:26:09 +0000] 62644 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-NAMENODE-format (log dir: None)
[12/Jan/2016 11:26:14 +0000] 62644 MainThread agent        INFO     PID '65336' associated with process '20-hdfs-NAMENODE-format' with payload 'processname:20-hdfs-NAMENODE-format groupname:20-hdfs-NAMENODE-format from_state:RUNNING expected:0 pid:65336' exited unexpectedly
[12/Jan/2016 11:26:14 +0000] 62644 MainThread agent        WARNING  Process exited event for process '20-hdfs-NAMENODE-format' does not have a monitor to handle event - the event will be ignored.

At this point I would be very grateful if someone could point me in the right direction.

Thanks

Re: Cloudera manager 5.5 install : HDFS error

rkalz — Tue, 12 Jan 2016 13:51:09 GMT

Found the issue, it was partially me not reading the log carefully enough.

The log from my first post mentions directory "/var/run/cloudera-scm-agent/process/20-hdfs-NAMENODE-format/logs", in which I found stderr.log with the lines :

Running in non-interactive mode, and data appears to exist in Storage Directory /soft/dfs/nn. Not formatting.
16/01/12 11:15:37 INFO util.ExitUtil: Exiting with status 1

It turns out there were files remaning from a previous attempt, cleaning them solved the issue.

Re: Cloudera manager 5.5 install : HDFS error

dennisli — Thu, 18 Jul 2019 09:17:47 GMT

yes me too, solved by cleanning the /dfs/nn and /dfs/dn

Re: Cloudera manager 5.5 install : HDFS error

SachinP — Wed, 15 Apr 2020 13:29:42 GMT

how to clean /dfs/nn from name node?

Re: Cloudera manager 5.5 install : HDFS error

dennisli — Thu, 07 May 2020 06:52:16 GMT

just rm -r the dir

question Re: Cloudera manager 5.5 install : HDFS error in Archives of Support Questions (Read Only)

Cloudera manager 5.5 install : HDFS error

Re: Cloudera manager 5.5 install : HDFS error

Re: Cloudera manager 5.5 install : HDFS error

Re: Cloudera manager 5.5 install : HDFS error

Re: Cloudera manager 5.5 install : HDFS error