question Re: How can yarn queue be specified for Titan mapreduce indexing jobs? in Archives of Support Questions (Read Only)

How can yarn queue be specified for Titan mapreduce indexing jobs?

bpgergo — Fri, 05 Jan 2018 18:33:24 GMT

whatever I tried, all yarn applications ended up in the default queue

these are the things I tried:

1) setting property in the titan-hbase-solr.properties (none of the following worked)

mapred.job.queue.name=myqueue

mapreduce.job.queue.name=myqueue

mapred.mapreduce.job.queue.name=myqueue

2) setting property in the gremlin shell

gremlin> graph = TitanFactory.open("/usr/iop/4.2.5.0-0000/titan/conf/titan-hbase-solr.properties")

gremlin> mgmt = graph.openManagement()

gremlin> desc = mgmt.getPropertyKey("desc2")

gremlin> mr = new MapReduceIndexManagement(graph)

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]:

hadoop gremlin> mgmt.set('hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: hadoop Display stack trace? [yN] n

gremlin> mgmt.set('titan.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: titan Display stack trace? [yN] n

gremlin> mgmt.set('mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapred Display stack trace? [yN] n gremlin>

gremlin> mgmt.set('mapreduce.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapreduce Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: mapred Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: hadoop Display stack trace? [yN] n gremlin>

Re: How can yarn queue be specified for Titan mapreduce indexing jobs?

yvora — Sat, 06 Jan 2018 03:41:50 GMT

@pbarna, you can set mapreduce.job.queuename=myqueue for mapred job.

https://community.hortonworks.com/content/supportkb/49658/how-to-specify-queue-name-submitting-mapreduce-job.html

Re: How can yarn queue be specified for Titan mapreduce indexing jobs?

amestry — Mon, 08 Jan 2018 03:03:42 GMT

@pbarna

I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved.

Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3.

My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect.

Example:

<property>
<name>mapreduce.local.map.tasks.maximum</name>
<value>10</value>
<final>true</final>
</property>

Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here.

Please feel free to reach out if you think this was useful.

Thanks @Nixon Rodrigues for letting me know about this question.