Support Questions
Find answers, ask questions, and share your expertise

HiveOnSpark not using cluster capacity completely at times

Cluster capacity: 60CPUS

Cluster Memory: 205 GB

This was not a problem until I was using Hive on MapReduce, However, sometimes, with HiveOnSpark, if my sqoop jobs are executing, CPU's usage and memory usage is not even half the available capacity yet my Hive jobs are waiting for it to finish off yet and it gets stuck sometimes.

I had to kill a workflow with sqoop jobs because it was stuck for 6hours and my Hive queries were dagging during this file.

Here's my configuration:

Spark Executor Cores: 1

spark.executor.memory : 5GB

spark.driver.memory : 256 MiB

spark.yarn.driver.memoryOverhead : 26MiB

spark.yarn.executor.memoryOverhead : 2GB