Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cloudera manager 5.11.0 can not kill balancer process

cloudera manager 5.11.0 can not kill balancer process

New Contributor

env:

cloudera manager 5.11.0

centos6.4   

cdh5.4.0

supervisor 3.0

 

 

balancer start with one_off_wrapper.sh

 

 

/bin/bash /usr/lib64/cmf/service/common/one_off_wrapper.sh /usr/lib64/cmf/service/hdfs/hdfs.sh balancer -threshold 1.0 -policy DataNode
hdfs     17593 15.2  2.8 2863588 705044 ?      Sl   14:36   0:09      \_ /usr/java/jdk1.8/bin/java -Dproc_balancer -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.hdfs.server.balancer.Balancer -threshold 1.0 -policy DataNode

 

When I stop balancer from website, the one_off_wrapper.sh killed, but java process remain.

 

supervisor conf

[program:706-hdfs-BALANCER]
command=/usr/lib64/cmf/service/common/one_off_wrapper.sh "/usr/lib64/cmf/service/hdfs/hdfs.sh" "balancer" "-threshold" "1.0" "-policy" "DataNode"
autostart=true
directory=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER
stdout_logfile=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER/logs/stdout.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
stderr_logfile=/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER/logs/stderr.log
stderr_logfile_maxbytes=10MB
stderr_logfile_backups=10
environment=CDH_PARQUET_HOME='/usr/lib/parquet',CDH_MR2_HOME='/usr/lib/hadoop-mapreduce',CMF_PACKAGE_DIR='/usr/lib64/cmf/service',CDH_HADOOP_BIN='/usr/bin/hadoop',MGMT_HOME='/usr/share/cmf',CDH_IMPALA_HOME='
/usr/lib/impala',CDH_YARN_HOME='/usr/lib/hadoop-yarn',CDH_KAFKA_HOME='/usr/lib/kafka',CDH_HDFS_HOME='/usr/lib/hadoop-hdfs',HOME='/var/lib/hadoop-hdfs',PATH='/sbin:/usr/sbin:/bin:/usr/bin',CDH_HUE_PLUGINS_HOM
E='/usr/lib/hadoop',PARCEL_DIRNAMES='CDH-5.4.0-1.cdh5.4.0.p0.27',CM_STATUS_CODES='STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MOD
E',KEYTRUSTEE_KP_HOME='/usr/share/keytrustee-keyprovider',CLOUDERA_ORACLE_CONNECTOR_JAR='/usr/share/java/oracle-connector-java.jar',CONF_DIR='/var/run/cloudera-scm-agent/process/706-hdfs-BALANCER',CDH_SQOOP2
_HOME='/usr/lib/sqoop2',KEYTRUSTEE_SERVER_HOME='/usr/lib/keytrustee-server',JSVC_HOME='/usr/libexec/bigtop-utils',HIVE_DEFAULT_XML='/etc/hive/conf.dist/hive-default.xml',JAVA_HOME='/usr/java/jdk1.8',CLOUDERA
_POSTGRESQL_JDBC_JAR='/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',PARCELS_ROOT='/home/cloudera/parcels',CDH_KMS_HOME='/usr/lib/hadoop-kms',CDH_HBASE_HOME='/usr/lib/hbase',CDH_SQOOP_HOME='/usr/lib/sqoop'
,WEBHCAT_DEFAULT_XML='/etc/hive-webhcat/conf.dist/webhcat-default.xml',CDH_OOZIE_HOME='/usr/lib/oozie',CDH_ZOOKEEPER_HOME='/usr/lib/zookeeper',CDH_HUE_HOME='/usr/lib/hue',CLOUDERA_MYSQL_CONNECTOR_JAR='/usr/s
hare/java/mysql-connector-java.jar',CDH_SPARK_HOME='/usr/lib/spark',HADOOP_BALANCER_OPTS='-Xms1073741824 -Xmx1073741824 ',CDH_MR1_HOME='/usr/lib/hadoop-0.20-mapreduce',TOMCAT_HOME='/usr/lib/bigtop-tomcat',CD
H_PIG_HOME='/usr/lib/pig',CDH_VERSION='5',HADOOP_ROOT_LOGGER='INFO,console',CDH_CRUNCH_HOME='/usr/lib/crunch',CDH_LLAMA_HOME='/usr/lib/llama/',CDH_HTTPFS_HOME='/usr/lib/hadoop-httpfs',CDH_HADOOP_HOME='/usr/l
ib/hadoop',CDH_HIVE_HOME='/usr/lib/hive',ORACLE_HOME='/usr/share/oracle/instantclient',CDH_HCAT_HOME='/usr/lib/hcatalog',CDH_SENTRY_HOME='/usr/lib/sentry',CDH_HBASE_INDEXER_HOME='/usr/lib/hbase-solr',SCM_DEF
INES_SCRIPTS='/home/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/meta/cdh_env.sh',CDH_SOLR_HOME='/usr/lib/solr',CDH_FLUME_HOME='/usr/lib/flume-ng'
user=hdfs
group=hdfs
exitcodes=0,2
autorestart=false
startsecs=0
killasgroup=true
# rlimits

 

agent log

 

[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Refreshing process files: None
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     prepare_environment begin: {u'PREPROCESS': u'V1.2', u'HDFSFIX': u'5.4.0-1.cdh5.4.0.p2', u'KAFKA': u'2.1.1-1.2.1.1.p0.18', u'CDH': u'5.4.0-1.
cdh5.4.0.p0.27', u'CUSTOM': u'5.4.0-1.cdh5.4.0.p2.3'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     The following requested parcels are not available: {}
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel PREPROCESS
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel HDFSFIX
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags [] for parcel CUSTOM
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     Obtained tags ['kafka'] for parcel KAFKA
[23/May/2017 14:15:12 +0000] 4950 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}
[23/May/2017 14:15:12 +0000] 4950 MainThread __init__     INFO     Extracted 6 files and 0 dirs to /var/run/cloudera-scm-agent/process/706-hdfs-BALANCER.
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Launching process. one-off True, command hdfs/hdfs.sh, args [u'balancer', u'-threshold', u'1.0', u'-policy', u'DataNode'
]
[23/May/2017 14:15:12 +0000] 4950 MainThread agent        INFO     Triggering supervisord update.
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Begin audit plugin refresh
[23/May/2017 14:15:12 +0000] 4950 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Begin metadata plugin refresh
[23/May/2017 14:15:12 +0000] 4950 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Not creating a monitor for 706-hdfs-BALANCER: should_monitor returns false
[23/May/2017 14:15:12 +0000] 4950 MainThread process      INFO     Daemon refresh complete for process 706-hdfs-BALANCER.
[23/May/2017 14:15:14 +0000] 4950 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:14 +0000] 4950 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:15:14 +0000] 4950 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:15:14 +0000] 4950 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:20:58 +0000] 4950 MainThread process      INFO     [706-hdfs-BALANCER] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[23/May/2017 14:20:58 +0000] 4950 MainThread process      INFO     Deactivating process 706-hdfs-BALANCER
[23/May/2017 14:20:59 +0000] 4950 MainThread agent        INFO     Triggering supervisord update.
[23/May/2017 14:20:59 +0000] 4950 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:20:59 +0000] 4950 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:20:59 +0000] 4950 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for hdfs-BALANCER with pipelines []
[23/May/2017 14:20:59 +0000] 4950 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hdfs-BALANCER (log dir: None)
[23/May/2017 14:24:59 +0000] 4950 MainThread heartbeat_tracker INFO     HB stats (seconds): num:43 LIFE_MIN:0.07 min:0.03 mean:0.07 max:0.21 LIFE_MAX:0.15

 

 

 

1 REPLY 1
Highlighted

Re: cloudera manager 5.11.0 can not kill balancer process

New Contributor

anyone has same problem?

 

Don't have an account?
Coming from Hortonworks? Activate your account here