Support Questions
Find answers, ask questions, and share your expertise

Region server not starting

Region server not starting

Explorer

stderr:   /var/lib/ambari-agent/data/errors-5640.txt

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase_regionserver.py", line 107, in status
    check_process_status(status_params.regionserver_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HBASE/package/scripts/hbase_regionserver.py", line 171, in <module>
    HbaseRegionServer().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1012, in restart
    self.post_start(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 424, in post_start
    raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file))
resource_management.core.exceptions.Fail: Pid file /var/run/hbase/hbase-hbase-regionserver.pid doesn't exist after starting of the component.

stdout:   /var/lib/ambari-agent/data/output-5640.txt

2020-05-14 02:32:10,000 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-14 02:32:10,020 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-14 02:32:10,266 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-14 02:32:10,277 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-14 02:32:10,280 - Skipping creation of User and Group as host is sys prepped or ignore_groupsusers_create flag is on
2020-05-14 02:32:10,280 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2020-05-14 02:32:10,283 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-05-14 02:32:10,285 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-05-14 02:32:10,286 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2020-05-14 02:32:10,349 - call returned (0, '1013')
2020-05-14 02:32:10,350 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1013'] {'not_if': '(test $(id -u hbase) -gt 1000) || (true)'}
2020-05-14 02:32:10,376 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1013'] due to not_if
2020-05-14 02:32:10,376 - Skipping setting dfs cluster admin and tez view acls as host is sys prepped
2020-05-14 02:32:10,376 - FS Type: HDFS
2020-05-14 02:32:10,376 - Directory['/etc/hadoop'] {'mode': 0755}
2020-05-14 02:32:10,402 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-05-14 02:32:10,403 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2020-05-14 02:32:10,423 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2020-05-14 02:32:10,553 - Skipping Execute[('setenforce', '0')] due to not_if
2020-05-14 02:32:10,554 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2020-05-14 02:32:10,556 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2020-05-14 02:32:10,557 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2020-05-14 02:32:10,557 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2020-05-14 02:32:10,562 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2020-05-14 02:32:10,563 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2020-05-14 02:32:10,570 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2020-05-14 02:32:10,582 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-05-14 02:32:10,582 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2020-05-14 02:32:10,583 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2020-05-14 02:32:10,588 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2020-05-14 02:32:10,593 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2020-05-14 02:32:10,599 - Skipping unlimited key JCE policy check and setup since it is not required
2020-05-14 02:32:11,017 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-05-14 02:32:11,028 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-05-14 02:32:11,033 - checked_call['hostid'] {}
2020-05-14 02:32:11,038 - checked_call returned (0, 'a8c03816')
2020-05-14 02:32:11,048 - Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-regionserver/conf stop regionserver'] {'only_if': 'ambari-sudo.sh  -H -E test -f /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh  -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1', 'on_timeout': '! ( ambari-sudo.sh  -H -E test -f /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh  -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1 ) || ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh  -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid`', 'timeout': 30, 'user': 'hbase'}
2020-05-14 02:32:11,055 - Skipping Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-regionserver/conf stop regionserver'] due to only_if
2020-05-14 02:32:11,056 - File['/var/run/hbase/hbase-hbase-regionserver.pid'] {'action': ['delete']}
2020-05-14 02:32:11,057 - Pid file /var/run/hbase/hbase-hbase-regionserver.pid is empty or does not exist
2020-05-14 02:32:11,076 - Directory['/etc/hbase'] {'mode': 0755}
2020-05-14 02:32:11,076 - Directory['/usr/hdp/current/hbase-regionserver/conf'] {'owner': 'hbase', 'group': 'hadoop', 'create_parents': True}
2020-05-14 02:32:11,077 - Directory['/tmp'] {'create_parents': True, 'mode': 0777}
2020-05-14 02:32:11,077 - Changing permission for /tmp from 1777 to 777
2020-05-14 02:32:11,077 - Directory['/tmp'] {'create_parents': True, 'cd_access': 'a'}
2020-05-14 02:32:11,078 - Execute[('chmod', '1777', u'/tmp')] {'sudo': True}
2020-05-14 02:32:11,085 - XmlConfig['hbase-site.xml'] {'owner': 'hbase', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hbase-regionserver/conf', 'configuration_attributes': {}, 'configurations': ...}
2020-05-14 02:32:11,098 - Generating config: /usr/hdp/current/hbase-regionserver/conf/hbase-site.xml
2020-05-14 02:32:11,098 - File['/usr/hdp/current/hbase-regionserver/conf/hbase-site.xml'] {'owner': 'hbase', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2020-05-14 02:32:11,153 - File['/usr/hdp/current/hbase-regionserver/conf/hdfs-site.xml'] {'action': ['delete']}
2020-05-14 02:32:11,153 - File['/usr/hdp/current/hbase-regionserver/conf/core-site.xml'] {'action': ['delete']}
2020-05-14 02:32:11,154 - XmlConfig['hbase-policy.xml'] {'owner': 'hbase', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hbase-regionserver/conf', 'configuration_attributes': {}, 'configurations': {u'security.admin.protocol.acl': u'*', u'security.masterregion.protocol.acl': u'*', u'security.client.protocol.acl': u'*'}}
2020-05-14 02:32:11,163 - Generating config: /usr/hdp/current/hbase-regionserver/conf/hbase-policy.xml
2020-05-14 02:32:11,163 - File['/usr/hdp/current/hbase-regionserver/conf/hbase-policy.xml'] {'owner': 'hbase', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2020-05-14 02:32:11,177 - File['/usr/hdp/current/hbase-regionserver/conf/hbase-env.sh'] {'content': InlineTemplate(...), 'owner': 'hbase', 'group': 'hadoop'}
2020-05-14 02:32:11,178 - Writing File['/usr/hdp/current/hbase-regionserver/conf/hbase-env.sh'] because contents don't match
2020-05-14 02:32:11,178 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2020-05-14 02:32:11,181 - File['/etc/security/limits.d/hbase.conf'] {'content': Template('hbase.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2020-05-14 02:32:11,182 - TemplateConfig['/usr/hdp/current/hbase-regionserver/conf/hadoop-metrics2-hbase.properties'] {'owner': 'hbase', 'template_tag': 'GANGLIA-RS'}
2020-05-14 02:32:11,191 - File['/usr/hdp/current/hbase-regionserver/conf/hadoop-metrics2-hbase.properties'] {'content': Template('hadoop-metrics2-hbase.properties-GANGLIA-RS.j2'), 'owner': 'hbase', 'group': None, 'mode': None}
2020-05-14 02:32:11,192 - TemplateConfig['/usr/hdp/current/hbase-regionserver/conf/regionservers'] {'owner': 'hbase', 'template_tag': None}
2020-05-14 02:32:11,194 - File['/usr/hdp/current/hbase-regionserver/conf/regionservers'] {'content': Template('regionservers.j2'), 'owner': 'hbase', 'group': None, 'mode': None}
2020-05-14 02:32:11,195 - Directory['/var/run/hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2020-05-14 02:32:11,195 - Directory['/var/log/hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2020-05-14 02:32:11,199 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2020-05-14 02:32:11,199 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-hbase.json
2020-05-14 02:32:11,199 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-hbase.json'] {'content': Template('input.config-hbase.json.j2'), 'mode': 0644}
2020-05-14 02:32:11,204 - File['/usr/hdp/current/hbase-regionserver/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hbase', 'group': 'hadoop', 'mode': 0644}
2020-05-14 02:32:11,205 - Package['phoenix_3_1_*'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2020-05-14 02:32:11,423 - Skipping installation of existing package phoenix_3_1_*
2020-05-14 02:32:11,423 - Ranger HBase plugin is not enabled
2020-05-14 02:32:11,424 - Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-regionserver/conf start regionserver'] {'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh  -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1', 'user': 'hbase'}

Command failed after 1 tries