Member since
02-03-2017
5
Posts
2
Kudos Received
0
Solutions
03-13-2017
09:25 PM
Responded in comment below.
... View more
03-13-2017
06:29 PM
/tmp is filling up on our hive-server2 node... I do not believe we have any sort of client/gateway node. We have beeline but do not use it, our app runs against hive directly. The /tmp/hive directory is because we have user=hive. The binary strings represent different datasets that we store in hadoop and query via hive. The vast majority of the disk space is taken up by a minority of datasets (This is an early check, the directory continues to fill until our 200gb disk runs out of space, when the job dies, and then cleans up the /tmp directory) root@hadoop-m-uscen-b-c001-n002:/tmp/hive# du -sh */
4.0K304dbcfe-6ba9-470f-9b2b-3c2f64f8d4eb/
7.1G72df6a01-9c1a-4de3-83be-9a239f86767f/
4.0Kf4dc7e2a-0cc9-415f-8d9e-e7115e0bbcea/
16Koperation_logs/
root@hadoop-m-uscen-b-c001-n002:/tmp/hive# du -sh */
4.0K304dbcfe-6ba9-470f-9b2b-3c2f64f8d4eb/
8.2G72df6a01-9c1a-4de3-83be-9a239f86767f/
4.0Kf4dc7e2a-0cc9-415f-8d9e-e7115e0bbcea/
... View more
03-13-2017
05:37 PM
Hi there,
We have a cluster with 2 hadoop namenodes and 3 datanodes. hive-server2 and hive-metastore are additionally running on each of the namenodes.
We're encountering an issue where particular jobs that are running seem to fill up the disk space on the local filesystem of the hive instance that our services are running against, causing jobs to fail and restart in an endless loop.
We initially had the configuration "hive.exec.local.scratchdir" set to /tmp/{user.name} and changed it to "hive.exec.scratchdir" in the hopes that that would cause it to write the tmp files to hdfs instead of a local directory with no success. We have now also set "hive.exec.mode.local.auto" to true per the configuration descriptions on this page: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties and all of the baseline criteria to 0, in an attempt to force hive not to run anything in local mode but still no success 😞 How can I prevent hive from writing its scratchdir/tmp files locally?? We do not have enough disk space on any single instance to hold all of the tmp data it wants to create.
... View more
Labels:
- Labels:
-
Apache Hive
03-01-2017
05:20 PM
That worked perfectly, thank you!
... View more
02-03-2017
09:58 PM
2 Kudos
Our instance of hiveserver2 has been running out of memory about once per day requiring a restart. The machines running hive serve2 have 52gb of memory, but is there some kind of rule of thumb on how much memory to allocate, based on machine size?? ps ax | grep -i hiveserver2 returns: hive 14668 1 0 Feb02 ? 00:02:16 /usr//bin/java -Xmx256m -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.7-4 -Dhdp.version=2.3.4.7-4 -Dhadoop.log.dir=/usr/hdp/2.3.4.7-4/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.7-4/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.4.7-4/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.7-4/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.7-4/hive/lib/hive-service-1.2.1.2.3.4.7-4.jar org.apache.hive.service.server.HiveServer2
... View more
Labels:
- Labels:
-
Apache Hive