error: [Errno 104] Connection reset by peer in agent log cloudera

[#1]

Hello All,

I noticed agent status bad notification for running 3-node cluster in cloudera manager CDH5.4 console.

When I tried to access the link for agent log, it gave me error.

I checked logs on the corresponding server at /var/log/cloudera-scm-agent , there I fiound out below error.

TP/1.1" 200 2120 "" "Java/1.7.0_67"
[08/Aug/2017 05:22:47 +0000] 8060 MainThread agent INFO Process with same id has changed: 2940-host-inspector.
[08/Aug/2017 05:22:47 +0000] 8060 MainThread agent INFO Deactivating process 2940-host-inspector
[08/Aug/2017 05:22:48 +0000] 8060 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-inspector with pipelines []
[08/Aug/2017 05:22:48 +0000] 8060 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for host-inspector (log dir: None)
[08/Aug/2017 05:22:49 +0000] 8060 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-inspector with pipelines []
[08/Aug/2017 05:22:49 +0000] 8060 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for host-inspector (log dir: None)
[08/Aug/2017 05:26:00 +0000] 8060 MonitorDaemon-Reporter throttling_logger ERROR (1 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-0fc5431c91f288535e98f1ed6d2d7836
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 74, in _send
self._requestor.request('sendAgentMessages', dict(messages=messages))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 139, in request
return self.issue_request(call_request, message_name, request_datum)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 249, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 478, in transceive
result = self.read_framed_message()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 482, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
response.begin()
File "/usr/lib64/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib64/python2.6/socket.py", line 433, in readline
data = recv(1)
error: [Errno 104] Connection reset by peer

Below are the outputs that I checked.

[root@LinuxUL cloudera-scm-agent]# cat /etc/hosts
127.0.0.1 localhost

10.68.200.34 LinuxUL.ad.infosys.com LinuxUL
10.68.200.152 linux152.ad.infosys.com linux152
10.68.200.170 linux170.ad.infosys.com linux170
172.21.5.224 nfrsat01.ad.infosys.com nfrsat01
10.67.200.77 blrsat06.ad.infosys.com blrsat06

[root@LinuxUL ~]# hostname -f
LinuxUL.ad.infosys.com
[root@LinuxUL ~]# python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
LinuxUL.ad.infosys.com 10.68.200.34

Also, selinux and iptables are disabled.

Please suggest.

Thanks,

Priya

[#2]

Hi cdhhadoop,

Cloudera agent is completely down? It happens in more servers?

Can you provide /var/log/cloudera-scm-agent/cloudera-scm-agent.out output?

Can you provide the output of the next commands?:

$ netstat -ltnp | grep :9000
$ source /etc/cloudera-scm-agent/config.ini &>/dev/null
$ ping -w1 $server_host
$ telnet $server_host $server_port

Regards,

Marc Casajus

[#3]

Hi marccasajus,

Seems like agent is completely down. It is on only one server as of now.
Command netstat -ltnp | grep :9000 gave no output.

[root@LinuxUL cloudera-scm-agent]# ping -w1 linux170.ad.infosys.com
PING linux170.ad.infosys.com (10.68.200.170) 56(84) bytes of data.
64 bytes from linux170.ad.infosys.com (10.68.200.170): icmp_seq=1 ttl=64 time=0.595 ms

--- linux170.ad.infosys.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.595/0.595/0.595/0.000 ms

[root@LinuxUL cloudera-scm-agent]# source /etc/cloudera-scm-agent/config.ini & > /dev/null
[1] 8911
[root@LinuxUL cloudera-scm-agent]# -bash: [General]: command not found
-bash: [Security]: command not found
-bash: [Hadoop]: command not found
-bash: [Cloudera]: command not found
-bash: [JDBC]: command not found

[1]+ Exit 127 source /etc/cloudera-scm-agent/config.ini
.
Please find /var/log/cloudera-scm-agent/cloudera-scm-agent.out output as below.


/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated
from sets import Set
[24/Mar/2017 20:12:05 +0000] 8060 MainThread agent INFO SCM Agent Version: 5.4.9
[24/Mar/2017 20:12:05 +0000] 8060 MainThread agent INFO Adding env vars that start with CMF_AGENT_
[24/Mar/2017 20:12:05 +0000] 8060 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
/usr/lib64/cmf/agent/src/cmf/agent.py:786: DeprecationWarning: psutil.cached_phymem is deprecated; use psutil.virtual_memory().cached instead
cached = psutil.cached_phymem()
/usr/lib64/cmf/agent/src/cmf/agent.py:787: DeprecationWarning: psutil.phymem_buffers is deprecated; use psutil.virtual_memory().buffers instead
buffers = psutil.phymem_buffers()
/usr/lib64/cmf/agent/src/cmf/agent.py:788: DeprecationWarning: psutil.used_phymem is deprecated; use psutil.phymem_usage().used instead
really_used = psutil.used_phymem() - cached - buffers
/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/psutil/__init__.py:1863: DeprecationWarning: psutil.phymem_usage is deprecated; use psutil.virtual_memory() instead
return phymem_usage().used
/usr/lib64/cmf/agent/src/cmf/monitor/host/__init__.py:190: DeprecationWarning: psutil.phymem_usage is deprecated; use psutil.virtual_memory() instead
really_used = psutil.phymem_usage().used - cached - buffers
2711-zookeeper-server: added process group
2711-zookeeper-server: stopped
2711-zookeeper-server: removed process group
2716-zookeeper-server: added process group
2720-hdfs-JOURNALNODE: added process group
2718-hdfs-DATANODE: added process group
2730-hbase-REGIONSERVER: added process group
2733-hue-HUE_SERVER: added process group
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh ']'
+ OLD_IFS='
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS='
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
+ . /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
+ env
2736-yarn-NODEMANAGER: added process group
2739-collect-host-statistics: added process group
2742-host-inspector: added process group
2746-solr-SOLR_SERVER: added process group
2733-hue-HUE_SERVER: stopped
2733-hue-HUE_SERVER: removed process group
2747-hue-HUE_SERVER: added process group
2739-collect-host-statistics: stopped
2739-collect-host-statistics: removed process group
2742-host-inspector: stopped
2742-host-inspector: removed process group
2749-collect-host-statistics: added process group
2752-host-inspector: added process group
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh ']'
+ OLD_IFS='
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS='
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
+ . /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
+ env
2756-mapreduce-TASKTRACKER: added process group
2752-host-inspector: stopped
2752-host-inspector: removed process group
2749-collect-host-statistics: stopped
2749-collect-host-statistics: removed process group
2761-collect-host-statistics: added process group
2764-host-inspector: added process group
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh ']'
+ OLD_IFS='
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS='
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
+ . /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
+ env
2736-yarn-NODEMANAGER: stopped
2736-yarn-NODEMANAGER: removed process group
2769-yarn-NODEMANAGER: added process group
2761-collect-host-statistics: stopped
2761-collect-host-statistics: removed process group
2764-host-inspector: stopped
2764-host-inspector: removed process group
2772-collect-host-statistics: added process group
2775-host-inspector: added process group
2772-collect-host-statistics: stopped
2772-collect-host-statistics: removed process group
2775-host-inspector: stopped
2775-host-inspector: removed process group
2778-kafka-KAFKA_BROKER: added process group
2779-collect-host-statistics: added process group
2782-host-inspector: added process group
2779-collect-host-statistics: stopped
2779-collect-host-statistics: removed process group
2782-host-inspector: stopped
2782-host-inspector: removed process group
2716-zookeeper-server: stopped
2716-zookeeper-server: removed process group
2785-zookeeper-server: added process group
2785-zookeeper-server: stopped
2785-zookeeper-server: removed process group
2788-zookeeper-server: added process group
2788-zookeeper-server: stopped
2788-zookeeper-server: removed process group
2793-zookeeper-server: added process group
2793-zookeeper-server: stopped
2793-zookeeper-server: removed process group
2796-zookeeper-server: added process group
2797-collect-host-statistics: added process group
2800-host-inspector: added process group
2797-collect-host-statistics: stopped
2797-collect-host-statistics: removed process group
2800-host-inspector: stopped
2800-host-inspector: removed process group
2803-collect-host-statistics: added process group
2806-host-inspector: added process group
2803-collect-host-statistics: stopped
2803-collect-host-statistics: removed process group
2806-host-inspector: stopped
2806-host-inspector: removed process group
2809-collect-host-statistics: added process group
2812-host-inspector: added process group
2809-collect-host-statistics: stopped
2809-collect-host-statistics: removed process group
2812-host-inspector: stopped
2812-host-inspector: removed process group
2796-zookeeper-server: stopped
2796-zookeeper-server: removed process group
2817-zookeeper-server: added process group
2817-zookeeper-server: stopped
2817-zookeeper-server: removed process group
2818-zookeeper-server: added process group
2818-zookeeper-server: stopped
2818-zookeeper-server: removed process group
2823-zookeeper-server: added process group
2823-zookeeper-server: stopped
2823-zookeeper-server: removed process group
2828-zookeeper-server: added process group
2829-collect-host-statistics: added process group
2832-host-inspector: added process group
2829-collect-host-statistics: stopped
2829-collect-host-statistics: removed process group
2832-host-inspector: stopped
2832-host-inspector: removed process group
2835-collect-host-statistics: added process group
2838-host-inspector: added process group
2835-collect-host-statistics: stopped
2835-collect-host-statistics: removed process group
2838-host-inspector: stopped
2838-host-inspector: removed process group
2842-collect-host-statistics: added process group
2845-host-inspector: added process group
2842-collect-host-statistics: stopped
2842-collect-host-statistics: removed process group
2845-host-inspector: stopped
2845-host-inspector: removed process group
2848-collect-host-statistics: added process group
2851-host-inspector: added process group
2848-collect-host-statistics: stopped
2848-collect-host-statistics: removed process group
2851-host-inspector: stopped
2851-host-inspector: removed process group
2730-hbase-REGIONSERVER: stopped
2730-hbase-REGIONSERVER: removed process group
2861-hbase-REGIONSERVER: added process group
2861-hbase-REGIONSERVER: stopped
2861-hbase-REGIONSERVER: removed process group
2867-hbase-REGIONSERVER: added process group
2878-collect-host-statistics: added process group
2881-host-inspector: added process group
2881-host-inspector: stopped
2881-host-inspector: removed process group
2878-collect-host-statistics: stopped
2878-collect-host-statistics: removed process group
2884-collect-host-statistics: added process group
2887-host-inspector: added process group
2887-host-inspector: stopped
2887-host-inspector: removed process group
2884-collect-host-statistics: stopped
2884-collect-host-statistics: removed process group
2890-collect-host-statistics: added process group
2893-host-inspector: added process group
2890-collect-host-statistics: stopped
2890-collect-host-statistics: removed process group
2893-host-inspector: stopped
2893-host-inspector: removed process group
2896-collect-host-statistics: added process group
2899-host-inspector: added process group
2896-collect-host-statistics: stopped
2896-collect-host-statistics: removed process group
2899-host-inspector: stopped
2899-host-inspector: removed process group
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh ']'
+ OLD_IFS='
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS='
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
+ . /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh
++ CDH_DIRNAME=CDH-5.4.0-1.cdh5.4.0.p0.27
++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-0.20-mapreduce
++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-hdfs
++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-httpfs
++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-mapreduce
++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-yarn
++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase
++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper
++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive
++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hue
++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/oozie
++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop
++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng
++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/pig
++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive-hcatalog
++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2
++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/llama
++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sentry
++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-tomcat
++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/bigtop-utils
++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/bin/hadoop
++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala
++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/solr
++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hbase-solr
++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/search
++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark
++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive-webhcat/conf.dist/webhcat-default.xml
++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop-kms
+ env
2769-yarn-NODEMANAGER: stopped
2769-yarn-NODEMANAGER: removed process group
2905-yarn-NODEMANAGER: added process group
2908-collect-host-statistics: added process group
2911-host-inspector: added process group
2908-collect-host-statistics: stopped
2908-collect-host-statistics: removed process group
2911-host-inspector: stopped
2911-host-inspector: removed process group
2914-collect-host-statistics: added process group
2917-host-inspector: added process group
2914-collect-host-statistics: stopped
2914-collect-host-statistics: removed process group
2917-host-inspector: stopped
2917-host-inspector: removed process group
2746-solr-SOLR_SERVER: stopped
2746-solr-SOLR_SERVER: removed process group
2921-solr-SOLR_SERVER: added process group
2921-solr-SOLR_SERVER: stopped
2921-solr-SOLR_SERVER: removed process group
2923-solr-SOLR_SERVER: added process group
2923-solr-SOLR_SERVER: stopped
2923-solr-SOLR_SERVER: removed process group
2925-solr-SOLR_SERVER: added process group
2926-collect-host-statistics: added process group
2929-host-inspector: added process group
2747-hue-HUE_SERVER: stopped
2747-hue-HUE_SERVER: removed process group
2932-hue-HUE_SERVER: added process group
2926-collect-host-statistics: stopped
2926-collect-host-statistics: removed process group
2929-host-inspector: stopped
2929-host-inspector: removed process group
2937-collect-host-statistics: added process group
2940-host-inspector: added process group
Segmentation fault (core dumped)

[#4]

Please run:

$ source /etc/cloudera-scm-agent/config.ini &>/dev/null
$ telnet $server_host $server_port

&> without any space between & and >.

Regards, 

Marc.

[#5]

Hi marccasajus,

source /etc/cloudera-scm-agent/config.ini &>/dev/null command gave no output.

Also please tell me what port should I use for telnet command?

Thanks,
Priya

[#6]

This command loads environment variables that you will use in the next command:

telnet $server_host $server_port

You need to check if the problem is for network issue or for application issue.

Regards,
Marc.

[#7]

Hi marccasajus,

I deleted the pid file and started the agent using sudo service cloudera-scm-agent start coommand and agent is now up and running. Seems like pid file was from older session.

Thanks for all the help.

Thanks,
priya

[#8]

Hi marccasajus,

Today again I saw the same error for agent as below.
sudo service cloudera-scm-agent status
cloudera-scm-agent dead but pid file exists

I see I can see 2937-collect-host-statistics: stopped
2937-collect-host-statistics: removed process group
2940-host-inspector: stopped
2940-host-inspector: removed process group in agent.out

And in agent.log, [25/Jun/2017 13:31:29 +0000] 8060 Monitor-SolrServerMonitor throttling_logger ERROR (52 skipped) Error fetching Solr core status at 'http://LinuxUL.ad.infosys.com:8983/solr//admin/cores?wt=json&action=STATUS'


Please help.

Thanks,
Priya

[#9]

Hi Team ,

can you please help on below error.

connection reset by peer error.

Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
19/01/31 18:43:18 INFO spark.SparkContext: Running Spark version 2.2.0.cloudera1
19/01/31 18:43:19 WARN spark.SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
19/01/31 18:43:19 INFO spark.SparkContext: Submitted application: Prime_CEP_BGFR_1309_Process Rates/Other Errors
19/01/31 18:43:19 INFO spark.SecurityManager: Changing view acls to: ggbmgphdpngrp
19/01/31 18:43:19 INFO spark.SecurityManager: Changing modify acls to: ggbmgphdpngrp
19/01/31 18:43:19 INFO spark.SecurityManager: Changing view acls groups to:
19/01/31 18:43:19 INFO spark.SecurityManager: Changing modify acls groups to:
19/01/31 18:43:19 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ggbmgphdpngrp); groups with view permissions: Set(); users with modify permissions: Set(ggbmgphdpngrp); groups with modify permissions: Set()
19/01/31 18:43:19 INFO util.Utils: Successfully started service 'sparkDriver' on port 50000.
19/01/31 18:43:19 INFO spark.SparkEnv: Registering MapOutputTracker
19/01/31 18:43:19 INFO spark.SparkEnv: Registering BlockManagerMaster
19/01/31 18:43:19 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/31 18:43:19 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/31 18:43:19 INFO storage.DiskBlockManager: Created local directory at /PBMG/users/ggbmgphdpngrp/prime/cep/tmp/blockmgr-88cc1ce5-d255-4009-9864-25e5f567879e
19/01/31 18:43:19 INFO memory.MemoryStore: MemoryStore started with capacity 6.2 GB
19/01/31 18:43:20 INFO spark.SparkEnv: Registering OutputCommitCoordinator
19/01/31 18:43:20 INFO util.log: Logging initialized @2402ms
19/01/31 18:43:20 INFO server.Server: jetty-9.3.z-SNAPSHOT
19/01/31 18:43:20 INFO server.Server: Started @2475ms
19/01/31 18:43:20 INFO server.AbstractConnector: Started ServerConnector@3ad394e6{HTTP/1.1,[http/1.1]}{0.0.0.0:52000}
19/01/31 18:43:20 INFO util.Utils: Successfully started service 'SparkUI' on port 52000.
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@26f143ed{/jobs,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@61a5b4ae{/jobs/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5b69fd74{/jobs/job,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77b325b3{/jobs/job/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e8e8651{/stages,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@271f18d3{/stages/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@61e3a1fd{/stages/stage,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@315df4bb{/stages/stage/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5cad8b7d{/stages/pool,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@25243bc1{/stages/pool/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2e6ee0bc{/storage,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@467f77a5{/storage/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@420bc288{/storage/rdd,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@308a6984{/storage/rdd/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7a34b7b8{/environment,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3be8821f{/environment/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b65e559{/executors,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@74a9c4b0{/executors/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c05a54d{/executors/threadDump,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5fd9b663{/executors/threadDump/json,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10567255{/static,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60b85ba1{/,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@117632cf{/api,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@159e366{/jobs/job/kill,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24528a25{/stages/stage/kill,null,AVAILABLE,@Spark}
19/01/31 18:43:20 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.6.209.22:52000
19/01/31 18:43:20 INFO spark.SparkContext: Added JAR file:/PBMG/users/ggbmgphdpngrp/prime/cep/prime-cep.jar at spark://10.6.209.22:50000/jars/prime-cep.jar with timestamp 1548956600301
19/01/31 18:43:20 INFO util.Utils: Using initial executors = 15, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
19/01/31 18:43:24 INFO yarn.Client: Requesting a new application from cluster with 8 NodeManagers
19/01/31 18:43:25 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (61440 MB per container)
19/01/31 18:43:25 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
19/01/31 18:43:25 INFO yarn.Client: Setting up container launch context for our AM
19/01/31 18:43:25 INFO yarn.Client: Setting up the launch environment for our AM container
19/01/31 18:43:25 INFO yarn.Client: Preparing resources for our AM container
19/01/31 18:43:25 INFO security.HadoopFSCredentialProvider: getting token for: hdfs://nameservice-np/user/ggbmgphdpngrp
19/01/31 18:43:25 INFO hdfs.DFSClient: Created token for ggbmgphdpngrp: HDFS_DELEGATION_TOKEN owner=ggbmgphdpngrp@BMEDIA.BAGINT.COM, renewer=yarn, realUser=, issueDate=1548956605075, maxDate=1549561405075, sequenceNumber=1281621, masterKeyId=1013 on ha-hdfs:nameservice-np
19/01/31 18:43:26 INFO hive.metastore: Trying to connect to metastore with URI thrift://gtunxlnu00853.server.arvato-systems.de:9083
19/01/31 18:43:26 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/01/31 18:43:26 INFO hive.metastore: Connected to metastore.
19/01/31 18:43:27 INFO metadata.Hive: Registering function dateconversion com.infosys.bmg.analytics.Date_Convert
19/01/31 18:43:27 INFO metadata.Hive: Registering function calc_week com.bmg.main.CalcWeek
19/01/31 18:43:27 INFO metadata.Hive: Registering function prev_week com.infosys.bmg.analytics.HiveUdfPrevWeek
19/01/31 18:43:27 INFO metadata.Hive: Registering function prev_week com.infosys.bmg.analytics.HiveUdfPrevWeek
19/01/31 18:43:27 INFO metadata.Hive: Registering function dateconversion com.infosys.bmg.analytics.Date_Convert
19/01/31 18:43:27 INFO metadata.Hive: Registering function date_convert com.infosys.bmg.analytics.Date_Convert
19/01/31 18:43:27 INFO metadata.Hive: Registering function calc_week com.infosys.bmg.analytics.HiveUdfCalcWeek
19/01/31 18:43:27 INFO metadata.Hive: Registering function day_of_week com.infosys.bmg.analytics.HiveUdfDayOfWeek
19/01/31 18:43:27 INFO metadata.Hive: Registering function beginning_of_fin_week_func com.infosys.bmg.date.Begining_Of_Financial_Week
19/01/31 18:43:27 INFO metadata.Hive: Registering function end_of_fin_week_func com.infosys.bmg.date.End_Of_Financial_Week
19/01/31 18:43:27 INFO metadata.Hive: Registering function dateconversion com.infosys.bmg.analytics.DateConvertFlash
19/01/31 18:43:27 INFO metadata.Hive: Registering function beginning_of_fin_week_func_ada com.infosys.bmg.date.BeginingOfFinancialWeekADA
19/01/31 18:43:27 INFO metadata.Hive: Registering function end_of_fin_week_func_ada com.infosys.bmg.date.EndOfFinancialWeekADA
19/01/31 18:43:27 INFO metadata.Hive: Registering function first_financial_day_func_ada com.infosys.bmg.date.FirstFinancialDayOfYearADA
19/01/31 18:43:27 INFO metadata.Hive: Registering function titleconversionudf com.infosys.bmg.Pr