Reply
New Contributor
Posts: 1
Registered: ‎03-31-2017

YARN web UI is not updated when using Hive on Spark

Hello:
I am trying to run Hive on Spark using Cloudera 5.7 docker, and I found no way of getting the job to show up in the YARN web UI. (0.0.0.0:8088)

The two setups tried are detailed above, one using Cloudera Manager and other without it.

WITH CLOUDERA MANAGER:


* Configured "Spark on YARN Service"
* Set spark as execution engine within Hive service configuration
* Started all the services

RESULTS:

Hive local_logs show the following lines, from which I understand that the queries were launched using spark as engine.
Despite this, they dont show in the YARN web UI.

2017-03-29 08:15:47,389 INFO  org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl: [main]: Setting up the session manager.
2017-03-29 08:15:47,444 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: loading spark properties from:spark-defaults.conf
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.shuffle.service.enabled -> true).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.eventLog.enabled -> true).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.yarn.jar -> local:/usr/lib/spark/lib/spark-assembly.jar).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.executor.extraLibraryPath -> /usr/lib/hadoop/lib/native).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.dynamicAllocation.minExecutors -> 0).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.dynamicAllocation.enabled -> true).
2017-03-29 08:15:47,445 INFO  org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory: [main]: load spark property from spark-defaults.conf (spark.master -> yarn-client).


WITHOUTH CLOUDERA MANAGER:
* Included the following settings in hive-site.xml:

hive.execution.engine=spark
spark.master=yarn-cluster
spark.executor.cores=1
spark.executor.instances=1
spark.eventLog.dir=/logs/
spark.eventLog.enabled=true

* Restarted hive service:
sudo service hive-server2 stop
sudo service hive-server2 start

RESULTS:

When using Cloudera Manager, hive logs dont show any lines regarding spark.
I connected to hive using beeline and confirmed that the values are properly setup by using, for instance:

set hive.execution.engine;


None of the setups reports back to Spark, and when configured without cloudera manager, Im not sure the proper engine is being used.

Could someone share any pointers towards what could be wrong within the described setups?

Thank you!

Highlighted
Cloudera Employee
Posts: 12
Registered: ‎12-14-2016

Re: YARN web UI is not updated when using Hive on Spark

Hi fperez,

 

A few questions that come to mind:

Are you getting results back from the job?

Just to confirm, is the Hive configuration, Spark On YARN Service, already set?

Does the job show up in Clusters > YARN Applications?


Cheers

Announcements