Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive demons are not getting started

Highlighted

Hive demons are not getting started

Expert Contributor

last week, Everything perfomed well for me.

I restarted My cluster after some edits I did today,

all service started just fine without any Problem except Hive how wouldn't start,

after a quick search I found that the problem comes from the disk space, this is the output of the error Message:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module>
    HiveMetastore().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 815, in restart
    self.stop(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 71, in stop
    hive_service('metastore', action='stop', upgrade_type=upgrade_type)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py", line 71, in hive_service
    pid = get_user_call_output.get_user_call_output(format("cat {pid_file}"), user=params.hive_user, is_checked_call=False)[1]
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 41, in get_user_call_output
    out_files.append(tempfile.NamedTemporaryFile())
  File "/usr/lib64/python2.6/tempfile.py", line 444, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner
    fd = _os.open(file, flags, 0600) 
OSError: [Errno 28] No space left on device: '/tmp/tmpqaafC3' 

i check out my disk to see how much space left, in this is screenshot of My filesystem: (df -h)

/dev/mapper/vg00-opt  2.9G  258M  2.5G  10% /opt 

/dev/mapper/vg00-tmp  2.0G  1.1G  796M  58% /tmp 

/dev/mapper/vg00-var  7.8G  3.7G  3.8G  50% /var

sounds the problem comes from other thing, because 800Mo is quite enough for hive, also hive is installed on 2 master nodes and they have the same space on /tmp

any suggestions will be greatly appreciated

1 REPLY 1

Re: Hive demons are not getting started

Contributor

@yassine sihi

From logs provided it clearly seems to be a disk related issue, as the device is getting filled up due to temp file getting created by Hive. You can try monitoring the /tmp space during the start of services which can provide you with more insight.

Small bash script can do this task for you :

#!/bin/bash
#Monitor space of temp
log_name="/tmp/tmp_data.log"
string="vg00-tmp"
dir="/tmp"
percent="80"

touch ${log_name}

while [ : ]
do
val=`df -h|grep ${string}|awk -F " " '{ print $5}'|sed 's/%//g'`
if [ ${val} -ge ${percent} ];then
        date >> ${log_name}
        printf "\n"
        du -kh ${dir}|sort -nr|head -1 1>>${log_name} 2>&1
        find ${dir} -size +20G 1>>${log_name} 2>&1
        printf " ~~~~~~~~~~~~~ \n" 1>>${log_name} 2>&1
        sleep 10
else
        sleep 10
fi
done
In the meanwhile, can you please provide us with some details regarding the changes which you have recently made? 
Don't have an account?
Coming from Hortonworks? Activate your account here