Support Questions
Find answers, ask questions, and share your expertise

cloudera manager 5.11.0 can not kill balancer process

Highlighted

cloudera manager 5.11.0 can not kill balancer process

New Contributor

env:

cloudera manager 5.11.0

centos6.4   

cdh5.4.0

supervisor 3.0

 

 

balancer start with one_off_wrapper.sh

 

 

/bin/bash /usr/lib64/cmf/service/common/one_off_wrapper.sh /usr/lib64/cmf/service/hdfs/hdfs.sh balancer -threshold 1.0 -policy DataNode
hdfs     17593 15.2  2.8 2863588 705044 ?      Sl   14:36   0:09      \_ /usr/java/jdk1.8/bin/java -Dproc_balancer -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.hdfs.server.balancer.Balancer -threshold 1.0 -policy DataNode

 

When I stop balancer from website, the one_off_wrapper.sh killed, but java process remain.

 

supervisor conf

[program:706-hdfs-BALANCER]
command=/usr/lib64/cmf/service/common/one_off_wrapper.sh "/usr/lib64/cmf/service/hdfs/hdfs.sh" "balancer" "-threshold" "1.0" "-policy" "DataNode"
autostart=true
directory=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER
stdout_logfile=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER/logs/stdout.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
stderr_logfile=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER/logs/stderr.log
stderr_logfile_maxbytes=10MB
stderr_logfile_backups=10
environment=CDH_PARQUET_HOME='/usr/lib/parquet',CDH_MR2_HOME='/usr/lib/hadoop-mapreduce',CMF_PACKAGE_DIR='/usr/lib64/cmf/service',CDH_HADOOP_BIN='/usr/bin/hadoop',MGMT_HOME='/usr/share/cmf',CDH_IMPALA_HOME='
/usr/lib/impala',CDH_YARN_HOME='/usr/lib/hadoop-yarn',CDH_KAFKA_HOME='/usr/lib/kafka',CDH_HDFS_HOME='/usr/lib/hadoop-hdfs',HOME='/var/lib/hadoop-hdfs',PATH='/sbin:/usr/sbin:/bin:/usr/bin',CDH_HUE_PLUGINS_HOM
E='/usr/lib/hadoop',PARCEL_DIRNAMES='CDH-5.4.0-1.cdh5.4.0.p0.27',CM_STATUS_CODES='STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MOD
E',KEYTRUSTEE_KP_HOME='/usr/share/keytrustee-keyprovider',CLOUDERA_ORACLE_CONNECTOR_JAR='/usr/share/java/oracle-connector-java.jar',CONF_DIR='/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER',CDH_SQOOP2
_HOME='/usr/lib/sqoop2',KEYTRUSTEE_SERVER_HOME='/usr/lib/keytrustee-server',JSVC_HOME='/usr/libexec/bigtop-utils',HIVE_DEFAULT_XML='/etc/hive/conf.dist/hive-default.xml',JAVA_HOME='/usr/java/jdk1.8',CLOUDERA
_POSTGRESQL_JDBC_JAR='/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',PARCELS_ROOT='/home/cloudera/parcels',CDH_KMS_HOME='/usr/lib/hadoop-kms',CDH_HBASE_HOME='/usr/lib/hbase',CDH_SQOOP_HOME='/usr/lib/sqoop'
,WEBHCAT_DEFAULT_XML='/etc/hive-webhcat/conf.dist/webhcat-default.xml',CDH_OOZIE_HOME='/usr/lib/oozie',CDH_ZOOKEEPER_HOME='/usr/lib/zookeeper',CDH_HUE_HOME='/usr/lib/hue',CLOUDERA_MYSQL_CONNECTOR_JAR='/usr/s
hare/java/mysql-connector-java.jar',CDH_SPARK_HOME='/usr/lib/spark',HADOOP_BALANCER_OPTS='-Xms1073741824 -Xmx1073741824 ',CDH_MR1_HOME='/usr/lib/hadoop-0.20-mapreduce',TOMCAT_HOME='/usr/lib/bigtop-tomcat',CD
H_PIG_HOME='/usr/lib/pig',CDH_VERSION='5',HADOOP_ROOT_LOGGER='INFO,console',CDH_CRUNCH_HOME='/usr/lib/crunch',CDH_LLAMA_HOME='/usr/lib/llama/',CDH_HTTPFS_HOME='/usr/lib/hadoop-httpfs',CDH_HADOOP_HOME='/usr/l
ib/hadoop',CDH_HIVE_HOME='/usr/lib/hive',ORACLE_HOME='/usr/share/oracle/instantclient',CDH_HCAT_HOME='/usr/lib/hcatalog',CDH_SENTRY_HOME='/usr/lib/sentry',CDH_HBASE_INDEXER_HOME='/usr/lib/hbase-solr',SCM_DEF
INES_SCRIPTS='/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh',CDH_SOLR_HOME='/usr/lib/solr',CDH_FLUME_HOME='/usr/lib/flume-ng'
user=hdfs
group=hdfs
exitcodes=0,2
autorestart=false
startsecs=0
killasgroup=true
# rlimits

 

agent log

 

[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Refreshing process files: None
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     prepare_environment begin: {u'PREPROCESS': u'V1.2', u'HDFSFIX': u'5.4.0-1.cdh5.4.0.p2', u'KAFKA': u'2.1.1-1.2.1.1.p0.18', u'CDH': u'5.4.0-1.
cdh5.4.0.p0.27', u'CUSTOM': u'5.4.0-1.cdh5.4.0.p2.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     The following requested parcels are not available: {}
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel PREPROCESS
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel HDFSFIX
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel CUSTOM
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags ['kafka'] for parcel KAFKA
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}
[23/May/2017 14:15:12 +0000] 4950 MainThread __init__     INFO     Extracted 6 files and 0 dirs to /var/run/cloudera-scm-agent/process/706-hdfs-BALANCER.
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Launching process. one-off True, command hdfs/hdfs.sh, args [u'balancer', u'-threshold', u'1.0', u'-policy', u'DataNode'
]
[23/May/2017 14:15:12 +0000] 4950 MainThread agent        INFO     Triggering supervisord update.
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Begin audit plugin refresh
[23/May/2017 14:15:12 +0000] 4950 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Begin metadata plugin refresh
[23/May/2017 14:15:12 +0000] 4950 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Not creating a monitor for 706-hdfs-BALANCER: should_monitor returns false
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Daemon refresh complete for process 706-hdfs-BALANCER.
[23/May/2017 14:15:14 +0000] 4950 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:14 +0000] 4950 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:15:14 +0000] 4950 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:14 +0000] 4950 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:20:58 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[23/May/2017 14:20:58 +0000] 4950 MainThread process      INFO     Deactivating process 706-hdfs-BALANCER
[23/May/2017 14:20:59 +0000] 4950 MainThread agent        INFO     Triggering supervisord update.
[23/May/2017 14:20:59 +0000] 4950 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:20:59 +0000] 4950 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:20:59 +0000] 4950 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:20:59 +0000] 4950 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:24:59 +0000] 4950 MainThread heartbeat_tracker INFO     HB stats (seconds): num:43 LIFE_MIN:0.07 min:0.03 mean:0.07 max:0.21 LIFE_MAX:0.15

 

 

 

1 REPLY 1
Highlighted

Re: cloudera manager 5.11.0 can not kill balancer process

New Contributor

anyone has same problem?

 

Don't have an account?