Support Questions
Find answers, ask questions, and share your expertise

Hive does not start on HDP 2.4 sandbox

Explorer

I am using HDP 2.4 sandbox. I have been using it for one month now, it was working fine. My laptop shutdown due to battery outage, and when I restart it, and start the sandbox, Hive does not fully start. Specifically, MYSQL Server fails to start, as shown below. Could someone tell me how to fix it?

Hive MetastoreStarted
HiveServer2Started
MySQL ServerStopped
WebHCat ServerStarted
Hive Client1 Hive Client Installed

Below are the details of the error messages reported in Ambari when I try to start MySQL Server:

/var/lib/ambari-agent/data/errors-1104.txt
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 64, in <module>
    MysqlServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 49, in start
    mysql_service(daemon_name=params.daemon_name, action='start')
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_service.py", line 42, in mysql_service
    sudo = True,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'service mysqld start' returned 1. MySQL Daemon failed to start.
Starting mysqld:  [FAILED]
/var/lib/ambari-agent/data/output-1104.txt
2016-07-11 12:53:28,489 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:28,489 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:28,490 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:28,536 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:28,536 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:28,579 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:28,579 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:28,579 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:28,767 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:28,767 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:28,767 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:28,805 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:28,805 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:28,847 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:28,848 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:28,848 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:28,851 - Group['hadoop'] {}
2016-07-11 12:53:28,860 - Group['users'] {}
2016-07-11 12:53:28,865 - Group['zeppelin'] {}
2016-07-11 12:53:28,865 - Group['knox'] {}
2016-07-11 12:53:28,865 - Group['ranger'] {}
2016-07-11 12:53:28,865 - Group['spark'] {}
2016-07-11 12:53:28,866 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,867 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,868 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,870 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,871 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,872 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,873 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,875 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']}
2016-07-11 12:53:28,876 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,877 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,878 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,879 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,880 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,881 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,884 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,885 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,885 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,887 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,888 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,889 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,891 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,892 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-07-11 12:53:28,895 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-07-11 12:53:28,902 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-07-11 12:53:28,903 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-07-11 12:53:28,904 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-07-11 12:53:28,906 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-07-11 12:53:28,913 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-07-11 12:53:28,914 - Group['hdfs'] {}
2016-07-11 12:53:28,914 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2016-07-11 12:53:28,916 - Directory['/etc/hadoop'] {'mode': 0755}
2016-07-11 12:53:28,948 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-07-11 12:53:28,949 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-07-11 12:53:28,984 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-07-11 12:53:29,016 - Skipping Execute[('setenforce', '0')] due to not_if
2016-07-11 12:53:29,016 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,020 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,021 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,029 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,032 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,033 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-07-11 12:53:29,051 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,051 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-07-11 12:53:29,053 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-07-11 12:53:29,060 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-07-11 12:53:29,065 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-07-11 12:53:29,363 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:29,363 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:29,363 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:29,413 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:29,413 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:29,464 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:29,464 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:29,464 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:29,579 - Execute[('service', 'mysqld', 'start')] {'logoutput': True, 'not_if': "pgrep -l '^mysqld", 'sudo': True}
MySQL Daemon failed to start. 
Starting mysqld:  [FAILED]


Below I have posted the content of /var/log/mysqld.log

[root@sandbox ~]# vi /var/log/mysqld.log

InnoDB: Doing recovery: scanned up to log sequence number 0 101200819

160711 13:09:48 InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 13:09:48 UTC - mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

key_buffer_size=8384512

read_buffer_size=131072

max_used_connections=0

max_threads=151

thread_count=0

connection_count=0

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338336 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x40000

/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x8509b9]

/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a3ef3]

/lib64/libpthread.so.0(+0xf7e0) [0x7f822e3017e0]

/usr/libexec/mysqld(page_cur_insert_rec_low+0x288) [0x799998]

/usr/libexec/mysqld(page_cur_parse_insert_rec+0x4eb) [0x79a53b]

/usr/libexec/mysqld() [0x7879ed]

/usr/libexec/mysqld(recv_recover_page+0x34f) [0x7894af]

/usr/libexec/mysqld(buf_page_io_complete+0x548) [0x74a868]

/usr/libexec/mysqld(fil_aio_wait+0xfa) [0x762a1a]

/usr/libexec/mysqld() [0x7c8a90]

/lib64/libpthread.so.0(+0x7aa1) [0x7f822e2f9aa1]

/lib64/libc.so.6(clone+0x6d) [0x7f822c9ec93d]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

160711 13:09:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

14 REPLIES 14

@Ebisa Negeri

Check if mysql is running using the below command:

service mysqld status

If not running, start the server as 'service mysqld start' and then try.

Thanks and Regards,

Sindhu

Explorer

Thanks Sindhu for your prompt reply. "service mysqld status" gave me "mysql is stopped". I get the failure result below when I try to start the server.

[root@sandbox ~]# service mysqld start

MySQL Daemon failed to start.

Starting mysqld: [FAILED]

Is there anything useful in the log file for mysql /var/log/mysqld.log

Can you ps and check if you can see mysqld running?

Does the pid file (/var/run/msqld/mysqld.pid) exist and if so what are the permissions on it? What is your OS?

Explorer

@awhitter Thank you for your response.

I have now posted the content of /var/log/mysqld.log at the bottom of the question.

mysqld is not running, and the /var/run/msqld/mysqld.pid does not exist. The directory /var/run/msqld is empty.

I am using HDP 2.4 sandbox, and uname -or gives me 2.6.32-573.18.1.el6.x86_64 GNU/Linux.

Any idea how to fix it?

Explorer

I am facing same issue.... Can anyone please suggest how to fix this? mysqld is not running, and the /var/run/msqld/mysqld.pid does not exist /var/run/msqld giving permission denied..

Thanks

Super Guru

i believe mysql was corripted. do a shutdown then a halt

on restart check http://dev.mysql.com/doc/refman/5.7/en/crashing.html

Explorer

Hi Timothy

Thanks... I tried this but mysqld is already stopped and not getting started.

I checked the attached link and I found /var/log//mysql/mysql.sock file doesn't exist, that's why admin command is not working..

Super Guru

anything in /var/log/mysql.err?

mysql.sock gets created when mysql starts successfully

are all the file permissions for the mysql directories ok

can you run mysqlcheck?

Explorer

/var/lib/mysql has 755 permission