question can we configure hive jobs not to run where spark is installed? in Archives of Support Questions (Read Only)

can we configure hive jobs not to run where spark is installed?

kjilla — Thu, 03 Mar 2016 01:34:52 GMT

How can we configure cluster to have spark separated from other echo system components

Re: can we configure hive jobs not to run where spark is installed?

aervits — Thu, 03 Mar 2016 01:42:22 GMT

You can logically segregate a cluster using Yarn node labels. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/ch_node_labels.html

You can also choose different queues for Spark and Hive. It won't necessarily prevent tasks running on same nodes but at least they won't compete for resources.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_performance_tuning/content/hive_perf_best_pract_better_wkld_mgmt_thru_queues.html

And http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/managing_cluster_capacity_with_queues.html

Re: can we configure hive jobs not to run where spark is installed?

kjilla — Thu, 03 Mar 2016 02:29:23 GMT

Thanks, that helps.

At the same time, can you point me to Setting Up Time-Based Queue Capacity Change.

Re: can we configure hive jobs not to run where spark is installed?

ravi1 — Thu, 03 Mar 2016 02:32:20 GMT

Its good it you separate this as a new question. Right now there is no support for time-based queue capacity change.

However, we were able to run a cron based job that refreshes queues with manual changes to capacity scheduler. However, if you do this and someone either restarts RMs and/or refreshes queues from ambari, your cron based changes will be overwritten.

Re: can we configure hive jobs not to run where spark is installed?

aervits — Thu, 03 Mar 2016 04:24:41 GMT

@kjilla if this is a satisfactory answer, please accept the answer.