Created on 03-24-2017 05:41 PM
SYMPTOMS: When using the SelectHiveQL processor in Nifi to run hive queries, it does not use the specific yarn queue ('nifi') if it is configured as in the following HiveConnectionPool settings:
jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;tez.queue.name=nifi
Following is an example of queue setting in yarn capacity scheduler:
yarn.scheduler.capacity.root.queues=default,nifi yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.maximum-capacity=40 yarn.scheduler.capacity.root.default.capacity=40 yarn.scheduler.capacity.root.default.acl_submit_applications=yarn yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.acl_administer_queue=yarn yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.queue-mappings-override.enable=true yarn.scheduler.capacity.root.default.acl_administer_queue=yarn yarn.scheduler.capacity.root.nifi.acl_administer_queue=* yarn.scheduler.capacity.root.nifi.acl_submit_applications=* yarn.scheduler.capacity.root.nifi.capacity=60 yarn.scheduler.capacity.root.nifi.maximum-capacity=60 yarn.scheduler.capacity.root.nifi.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.nifi.ordering-policy=fifo yarn.scheduler.capacity.root.nifi.state=RUNNING yarn.scheduler.capacity.root.nifi.user-limit-factor=1
RESOLUTION: Please add tez.queue.name with a question mark like this: ?tez.queue.name=<queue_name>
For example:
jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;?tez.queue.name=nifi