Community Articles

Find and share helpful community-sourced technical articles.
Labels (2)
avatar
Expert Contributor

SYMPTOMS: When using the SelectHiveQL processor in Nifi to run hive queries, it does not use the specific yarn queue ('nifi') if it is configured as in the following HiveConnectionPool settings:

jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;tez.queue.name=nifi

Following is an example of queue setting in yarn capacity scheduler:

yarn.scheduler.capacity.root.queues=default,nifi
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.maximum-capacity=40
yarn.scheduler.capacity.root.default.capacity=40
yarn.scheduler.capacity.root.default.acl_submit_applications=yarn
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=yarn
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.default.acl_administer_queue=yarn
yarn.scheduler.capacity.root.nifi.acl_administer_queue=*
yarn.scheduler.capacity.root.nifi.acl_submit_applications=*
yarn.scheduler.capacity.root.nifi.capacity=60
yarn.scheduler.capacity.root.nifi.maximum-capacity=60
yarn.scheduler.capacity.root.nifi.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.nifi.ordering-policy=fifo
yarn.scheduler.capacity.root.nifi.state=RUNNING
yarn.scheduler.capacity.root.nifi.user-limit-factor=1

RESOLUTION: Please add tez.queue.name with a question mark like this: ?tez.queue.name=<queue_name>

For example:

jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;?tez.queue.name=nifi
1,543 Views
0 Kudos