Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while starting Datanode from Ambari..! Error while executing command 'start':

Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

Hello,

I restarted the datanode from Ambari console and now it is failing to start with the below error:

2016-09-28 12:07:33,072 - Error while executing command 'start':
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/datanode.py", line 37, in start
    datanode(action="start")
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/hdfs_datanode.py", line 55, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/utils.py", line 102, in service
    not_if=service_is_up
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 241, in action_run
    raise ex
Fail: Execution of 'ulimit -c unlimited;  su -s /bin/bash - hdfs -c 'export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-genome-prod4.axs.out
Usage:
  hdfs [-a ALIAS] [--info] [-j] [-d DEPTH] [RPATH]
  hdfs [-a ALIAS] --read RPATH
  hdfs [-a ALIAS] --write [-o] RPATH
  hdfs [-a ALIAS] --download [-o] [-t THREADS] RPATH LPATH 

hdfs -h | --help | -l | --log | -v | --version

I check the hadoop-hdfs-datanode-genome-prod4.axs.log, but there is no error logging.

Only the '.out' file is getting updated with

Usage:
  hdfs [-a ALIAS] [--info] [-j] [-d DEPTH] [RPATH]
  hdfs [-a ALIAS] --read RPATH
  hdfs [-a ALIAS] --write [-o] RPATH
  hdfs [-a ALIAS] --download [-o] [-t THREADS] RPATH LPATH 

Read about the namenode format option on "stackoverflow", but I can't do that in a production environment. I tried to run the ulimit and start datanode commands from the command line, but nothing is happening.

Need help to get the datanode started.

9 REPLIES 9

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

@Suyog Nagaokar

What you get when you run below command?

ulimit -c unlimited; su -s /bin/bash - hdfs -c 'export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode'

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-genome-prod4.axs.out Usage: hdfs [-a ALIAS] [--info] [-j] [-d DEPTH] [RPATH] hdfs [-a ALIAS] --read RPATH hdfs [-a ALIAS] --write [-o] RPATH hdfs [-a ALIAS] --download [-o] [-t THREADS] RPATH LPATH hdfs -h | --help | -l | --log | -v | --version

but the datanode doesn't start..!!

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

output of cat /var/log/hadoop/hdfs/hadoop-hdfs-datanode-genome-prod4.axs.out :

Usage: hdfs [-a ALIAS] [--info] [-j] [-d DEPTH] [RPATH] hdfs [-a ALIAS] --read RPATH hdfs [-a ALIAS] --write [-o] RPATH hdfs [-a ALIAS] --download [-o] [-t THREADS] RPATH LPATH hdfs -h | --help | -l | --log | -v | --version ulimit -a for user hdfs core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515009 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

Hi Rahul Pathak, I have posted the command output, can you please guide me further ?

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

@ suyog Nagokar,

Please check /var/log/ambari-server/datanode logs.

and paste here.

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

Sorry for late reply...

I tried started the datanode again today and got the below logs in ambari-server.log:

16:25:37,312  INFO [qtp1523673911-915157] AmbariManagementControllerImpl:2173 - Received a updateHostComponent request, clusterName=************_Cluster, serviceName=HDFS, componentName=DATANODE, hostname=********.com, request={ clusterName=**********, serviceName=HDFS, componentName=DATANODE, hostname=**********.com, desiredState=STARTED, desiredStackId=null, staleConfig=null, adminState=null}
16:25:37,317  INFO [qtp1523673911-915157] RoleCommandOrder:220 - Role command order info was loaded from file: /var/lib/ambari-server/resources/stacks/HDP/2.2/role_command_order.json
16:25:37,350  INFO [qtp1523673911-915163] PersistKeyValueService:82 - Looking for keyName admin-settings-show-bg-admin
16:25:37,369  INFO [qtp1523673911-915189] PersistKeyValueService:82 - Looking for keyName admin-settings-show-bg-admin
16:25:46,247  WARN [qtp466462723-915165] HeartBeatHandler:411 - Operation failed - may be retried. Service component host: DATANODE, host: **********.com Action id787-1

On that node's ambari-agent.log, below is the error:

Error while executing command \'start\':\nTraceback (most recent call last):\n  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute\n    method(env)\n  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/datanode.py", line 37, in start\n    datanode(action="start")\n  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/hdfs_datanode.py", line 55, in datanode\n    create_log_dir=True\n  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/utils.py", line 102, in service\n    not_if=service_is_up\n  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__\n    self.env.run()\n  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run\n    self.run_action(resource, action)\n  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action\n    provider_action()\n  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 241, in action_run\n    raise ex\nFail: Execution of \'ulimit -c unlimited;  su -s /bin/bash - hdfs -c \'export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode\'\' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-*********.out

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

FYI @Ashnee Sharma

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

Explorer

I am also facing same issue. Is issue resolved @Suyog Nagaokar .?

Thanks & Regards
Aggy

Re: Error while starting Datanode from Ambari..! Error while executing command 'start':

New Contributor

Hi Vinod,

Can you check the cluster-id on the namenode and the datanode if there is any mismatch...?

Don't have an account?
Coming from Hortonworks? Activate your account here