Support Questions
Find answers, ask questions, and share your expertise

Hive does not start on HDP 2.4 sandbox

Explorer

I am using HDP 2.4 sandbox. I have been using it for one month now, it was working fine. My laptop shutdown due to battery outage, and when I restart it, and start the sandbox, Hive does not fully start. Specifically, MYSQL Server fails to start, as shown below. Could someone tell me how to fix it?

Hive MetastoreStarted
HiveServer2Started
MySQL ServerStopped
WebHCat ServerStarted
Hive Client1 Hive Client Installed

Below are the details of the error messages reported in Ambari when I try to start MySQL Server:

/var/lib/ambari-agent/data/errors-1104.txt
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 64, in <module>
    MysqlServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_server.py", line 49, in start
    mysql_service(daemon_name=params.daemon_name, action='start')
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/mysql_service.py", line 42, in mysql_service
    sudo = True,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'service mysqld start' returned 1. MySQL Daemon failed to start.
Starting mysqld:  [FAILED]
/var/lib/ambari-agent/data/output-1104.txt
2016-07-11 12:53:28,489 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:28,489 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:28,490 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:28,536 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:28,536 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:28,579 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:28,579 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:28,579 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:28,767 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:28,767 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:28,767 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:28,805 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:28,805 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:28,847 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:28,848 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:28,848 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:28,851 - Group['hadoop'] {}
2016-07-11 12:53:28,860 - Group['users'] {}
2016-07-11 12:53:28,865 - Group['zeppelin'] {}
2016-07-11 12:53:28,865 - Group['knox'] {}
2016-07-11 12:53:28,865 - Group['ranger'] {}
2016-07-11 12:53:28,865 - Group['spark'] {}
2016-07-11 12:53:28,866 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,867 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,868 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,870 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,871 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,872 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,873 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,875 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']}
2016-07-11 12:53:28,876 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,877 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,878 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,879 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,880 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,881 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,884 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,885 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2016-07-11 12:53:28,885 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,887 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,888 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,889 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,891 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2016-07-11 12:53:28,892 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-07-11 12:53:28,895 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-07-11 12:53:28,902 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-07-11 12:53:28,903 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-07-11 12:53:28,904 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-07-11 12:53:28,906 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-07-11 12:53:28,913 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-07-11 12:53:28,914 - Group['hdfs'] {}
2016-07-11 12:53:28,914 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2016-07-11 12:53:28,916 - Directory['/etc/hadoop'] {'mode': 0755}
2016-07-11 12:53:28,948 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-07-11 12:53:28,949 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-07-11 12:53:28,984 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-07-11 12:53:29,016 - Skipping Execute[('setenforce', '0')] due to not_if
2016-07-11 12:53:29,016 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,020 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,021 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-07-11 12:53:29,029 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,032 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,033 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-07-11 12:53:29,051 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-07-11 12:53:29,051 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-07-11 12:53:29,053 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2016-07-11 12:53:29,060 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-07-11 12:53:29,065 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-07-11 12:53:29,363 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-07-11 12:53:29,363 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-07-11 12:53:29,363 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-07-11 12:53:29,413 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-07-11 12:53:29,413 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-07-11 12:53:29,464 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-07-11 12:53:29,464 - Ensuring that hadoop has the correct symlink structure
2016-07-11 12:53:29,464 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-07-11 12:53:29,579 - Execute[('service', 'mysqld', 'start')] {'logoutput': True, 'not_if': "pgrep -l '^mysqld", 'sudo': True}
MySQL Daemon failed to start. 
Starting mysqld:  [FAILED]


Below I have posted the content of /var/log/mysqld.log

[root@sandbox ~]# vi /var/log/mysqld.log

InnoDB: Doing recovery: scanned up to log sequence number 0 101200819

160711 13:09:48 InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 13:09:48 UTC - mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

key_buffer_size=8384512

read_buffer_size=131072

max_used_connections=0

max_threads=151

thread_count=0

connection_count=0

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338336 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x40000

/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x8509b9]

/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a3ef3]

/lib64/libpthread.so.0(+0xf7e0) [0x7f822e3017e0]

/usr/libexec/mysqld(page_cur_insert_rec_low+0x288) [0x799998]

/usr/libexec/mysqld(page_cur_parse_insert_rec+0x4eb) [0x79a53b]

/usr/libexec/mysqld() [0x7879ed]

/usr/libexec/mysqld(recv_recover_page+0x34f) [0x7894af]

/usr/libexec/mysqld(buf_page_io_complete+0x548) [0x74a868]

/usr/libexec/mysqld(fil_aio_wait+0xfa) [0x762a1a]

/usr/libexec/mysqld() [0x7c8a90]

/lib64/libpthread.so.0(+0x7aa1) [0x7f822e2f9aa1]

/lib64/libc.so.6(clone+0x6d) [0x7f822c9ec93d]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

160711 13:09:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

14 REPLIES 14

Explorer

@Timothy Spann

Yes.. mySQLcheck is running...

6045-2016-07-25-2.png

Mentor

Check the space for MySQL data directory, check whether MySQL port is being used by something else. check whether permissions have changed

du -hs /path/*
df -hs *
netstat -tulnp / grep 3306
ls -ltra /path

Super Guru

@Swati Gupta / @Ebisa Negeri

Can you please try below steps:

# cd /var/lib/mysql
# mkdir bak
# mv ibdata1 bak/.
# mv ib_logfile* bak/.
# cp -a bak/ibdata1 ibdata1
# cp -a bak/ib_logfile* .
# service mysql restart

Reference - http://notesonit.blogspot.hk/2013/05/innodb-unable-to-lock-ibdata1-error-11.html

Explorer

I am facing this same problem. Did you find any solution to this problem?

Expert Contributor

http://notesonit.blogspot.hk/2013/05/innodb-unable-to-lock-ibdata1-error-11.html , see if the steps in this link work for you. I see that you also got the same error #11 from your logs. put the steps below for convenience

I was able to fix the problem by restoring the permissions on several files (modified from this post): # cd /var/lib/mysql # mkdir bak # mv ibdata1 bak/. # mv ib_logfile* bak/. # cp -a bak/ibdata1 ibdata1 # cp -a bak/ib_logfile* .

# service mysql restart