Reply
Explorer
Posts: 14
Registered: ‎04-15-2015

failed to start resourcemanager in the process of installing YARN with cdh5.3.3

thanks of your help,here is the detail: 1、the error message is : Failed to execute command Start on service YARN (MR2 Included)。 2、and there is nothing in resourcemanager log. /var/log/hadoop-yarn/hadoop-cmf-yarn-RESOURCEMANAGER-hnfdpnn01.log.out
Cloudera Employee
Posts: 12
Registered: ‎03-31-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

Are you using Cloudera Manager?

Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

Yeah,The CM edition is cm-5.3.3,and install CDH refering the following method:
Installation Path C - Manual Installation Using Cloudera Manager Tarballs
Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

By the way,the nodemanager was started normally
Cloudera Employee
Posts: 12
Registered: ‎03-31-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

OK, couple of things to check. Is the CM agent started and heartbeating on that server? Also please check the stdout and stderr logs which are alongside the role log in CM for that service.

Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

thanks for your patience,the CM agent started heartbeating normally ,but there is error message with CM agent log (cloudera-scm-agent.out):
[15/Apr/2015 23:09:11 +0000] 22578 MainThread util INFO Extracted 20 files and 0 dirs to /opt/cm-5.3.3/run/cloudera-scm-
agent/process/331-yarn-RESOURCEMANAGER.
[15/Apr/2015 23:09:11 +0000] 22578 MainThread util ERROR Failed to parse environment variables: CDH_HCAT_HOME=/opt/cloude
ra/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hive-hcatalog
TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/bigtop-tomcat
HOSTNAME=hnfdpnn01
YARN_RESOURCEMANAGER_OPTS=-Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMS
InitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dhadoop.event.appender=,EventCatcher -XX:OnOutOfMemoryError=/opt/cm-5.
3.3/lib64/cmf/service/common/killparent.sh
CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/pig
CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/solr
PARCELS_ROOT=/opt/cloudera/parcels
TERM=vt100
CMF_AGENT_MGMT_HOME=/opt/cm-5.3.3/share/cmf
CLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
SHELL=/bin/bash
HISTSIZE=4096
SSH_CLIENT=10.209.18.150 64714 22
CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop
CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hive
SCM_DEFINES_SCRIPTS=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/meta/cdh_env.sh
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
SEARCH_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/search
CLOUDERA_ORACLE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java.jar
SSH_TTY=/dev/pts/0
HIVE_DEFAULT_XML=/etc/hive/conf.dist/hive-default.xml
CDH_VERSION=5
MGMT_HOME=/opt/cm-5.3.3/share/cmf
CMF_CONF_DIR=/opt/cm-5.3.3/etc/cloudera-scm-agent
CMF_SUDO_CMD=
PARCEL_DIRNAMES=CDH-5.3.3-1.cdh5.3.3.p0.5
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;
43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31
:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=0
1;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;
35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=0
1;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm
=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc
=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=0
1;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=
01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
CDH_SQOOP_HOME=/usr/lib/sqoop
JSVC_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/bigtop-utils
SSH_AUTH_SOCK=/tmp/ssh-8gUQf06XcK/agent.19562
CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/impala
CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-mapreduce
CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-httpfs
TMOUT=0
CMF_SBINDIR=/opt/cm-5.3.3/sbin
CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-0.20-mapreduce
CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hue
CM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_ST
ANDBY_MODE
YARN_ROOT_LOGGER=INFO,RFA
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/kerberos/bin
MAIL=/var/spool/mail/root
CONF_DIR=/opt/cm-5.3.3/run/cloudera-scm-agent/process/331-yarn-RESOURCEMANAGER
CMF_SERVER_ROOT=/opt/cm-5.3.3/share/cmf
CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/sentry
CDH_PARQUET_HOME=/usr/lib/parquet
PWD=/root
JAVA_HOME=/usr/java/jdk1.7.0_67/
KDE_IS_PRELINKED=1
LANG=zh_CN.gbk
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop
LOADEDMODULES=
KDEDIRS=/usr
ROOT=/opt/cm-5.3.3/lib64/cmf
CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/sqoop2
CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark
CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/llama
YARN_LOG_DIR=/var/log/hadoop-yarn
CDH_CRUNCH_HOME=/usr/lib/crunch
KRB5CCNAME=/tmp/krb5cc_cm_agent_0
HISTCONTROL=ignoredups
CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-hdfs
CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/bin/hadoop
SHLVL=4
HOME=/root
CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/oozie
YARN_LOGFILE=hadoop-cmf-yarn-RESOURCEMANAGER-hnfdpnn01.log.out
CMF_AGENT_ROOT=/opt/cm-5.3.3/lib64/cmf
CMF_ETC=/opt/cm-5.3.3/etc
CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hbase-solr
CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/flume-ng
KEYTRUSTEE_KP_HOME=/usr/share/keytrustee-keyprovider
LOGNAME=root
CVS_RSH=ssh
QTLIB=/usr/lib64/qt-3.3/lib
SSH_CONNECTION=10.209.18.150 64714 134.160.36.45 22
CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hbase
CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-kms
MODULESHOME=/usr/share/Modules
CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop-yarn
CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/zookeeper
LESSOPEN=|/usr/bin/lesspipe.sh %s
CMF_VAR=/opt/cm-5.3.3
CMF_PACKAGE_DIR=/opt/cm-5.3.3/lib64/cmf/service
WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/etc/hive-webhcat/conf.dist/webhcat-default.xml
CLOUDERA_POSTGRESQL_JDBC_JAR=/opt/cm-5.3.3/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
G_BROKEN_FILENAMES=1
HISTFILE=/tmp/.hist/root/10.209.18.150.his.20150415_194701
BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*`
}
_=/bin/env
Traceback (most recent call last):
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/util.py", line 362, in source
return dict((line.split("=", 1) for line in data.splitlines()))
ValueError: dictionary update sequence element #91 has length 1; 2 is required
[15/Apr/2015 23:09:11 +0000] 22578 MainThread agent ERROR Failed to activate {u'refresh_files': [u'cloudera-stack-monitor.
properties', u'capacity-scheduler.xml', u'fair-scheduler.xml', u'nodes_allow.txt', u'nodes_exclude.txt', u'topology.map'], u'config_
generation': 0, u'auto_restart': False, u'running': True, u'required_tags': [u'cdh'], u'user': u'yarn', u'special_file_info': [{u'pe
rmissions': 420, u'deploy_dir': u'/etc/hadoop/conf.cloudera.yarn', u'group': u'hadoop', u'name': u'topology.map', u'user': u'root'},
{u'permissions': 493, u'deploy_dir': u'/etc/hadoop/conf.cloudera.yarn', u'group': u'hadoop', u'name': u'topology.py', u'user': u'ro
ot'}], u'group': u'hadoop', u'id': 331, u'status_links': {u'status': u'http://hnfdpnn01:8088/'}, u'name': u'yarn-RESOURCEMANAGER', u
'run_generation': 1, u'environment': {u'YARN_RESOURCEMANAGER_OPTS': u'-Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMar
kSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dhadoop.event.appender=,Ev
entCatcher -XX:OnOutOfMemoryError={{AGENT_COMMON_DIR}}/killparent.sh', u'YARN_LOGFILE': u'hadoop-cmf-yarn-RESOURCEMANAGER-hnfdpnn01.
log.out', u'YARN_ROOT_LOGGER': u'INFO,RFA', u'YARN_LOG_DIR': u'/var/log/hadoop-yarn', u'CDH_VERSION': u'5'}, u'optional_tags': [u'cd
h-plugin', u'yarn-plugin', u'mapreduce2-plugin'], u'program': u'yarn/yarn.sh', u'arguments': [u'resourcemanager'], u'parcels': {u'CD
H': u'5.3.3-1.cdh5.3.3.p0.5'}, u'resources': [{u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits':
{u'limit_memlock': None, u'limit_fds': None}, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'nam
ed_cpu': None, u'tcp_listen': {u'bind_address': u'134.160.36.45', u'port': 8032}, u'dynamic': False, u'rlimits': None, u'file': None
, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'134.160.3
6.45', u'port': 8030}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io
': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'134.160.36.45', u'port': 8031}, u'dynamic': False, u'rlimits': None,
u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address
': u'134.160.36.45', u'port': 8033}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu'
: None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'134.160.36.45', u'port': 8088}, u'dynamic': False, u'r
limits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': N
one, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/hadoop-yarn', u'bytes_f
ree_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': N
one, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/had
oop-yarn/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 493}, u'cpu': None}], u
'one_off': False}
Traceback (most recent call last):
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/agent.py", line 1234, in handle_heartbeat_response
new_process.activate()
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/agent.py", line 2441, in activate
self.write_process_conf()
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/agent.py", line 2551, in write_process_conf
"source_parcel_environment", env))
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/util.py", line 365, in source
raise e
ValueError: dictionary update sequence element #91 has length 1; 2 is required
[15/Apr/2015 23:09:13 +0000] 22578 Audit-Plugin audit_plugin INFO stopping Audit Plugin with pipelines
[15/Apr/2015 23:09:13 +0000] 22578 Audit-Plugin audit_plugin_pipeline INFO stopping (None)
Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

Stderr:
HTTP ERROR 403

Problem accessing /cmf/process/331/logs. Reason:
http://hnfdpnn01:9000/process/331-yarn-RESOURCEMANAGER/files/logs/stderr.log
?????????????
Stdout:
HTTP ERROR 403

Problem accessing /cmf/process/331/logs. Reason:
http://hnfdpnn01:9000/process/331-yarn-RESOURCEMANAGER/files/logs/stdout.log
?????????????
Cloudera Employee
Posts: 12
Registered: ‎03-31-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

Thanks for the logs. I suggest to try restarting the agent, but ensure you use "service cloudera-scm-agent start" rather than "/etc/init.d/cloudera-scm-agent start" so that settings from your environment don't leak into the script. 

Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

i installed CM agent with tarbar,you mean i should register CM agent with os system?
is there another way instead?
Explorer
Posts: 14
Registered: ‎04-15-2015

Re: failed to start resourcemanager in the process of installing YARN with cdh5.3.3

there is also error message with CM agent log ,is that matter?
:[15/Apr/2015 23:50:49 +0000] 11327 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose:
mgmt-HOSTMONITOR-05e3a3aa1adff1e6c5935c455391d199
Traceback (most recent call last):
File "/opt/cm-5.3.3/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 70, in _send
self._port)
File "/opt/cm-5.3.3/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 464, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
Announcements