Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can yarn queue be specified for Titan mapreduce indexing jobs?

avatar
Rising Star

whatever I tried, all yarn applications ended up in the default queue

these are the things I tried:

1) setting property in the titan-hbase-solr.properties (none of the following worked)

mapred.job.queue.name=myqueue

mapreduce.job.queue.name=myqueue

mapred.mapreduce.job.queue.name=myqueue

2) setting property in the gremlin shell

gremlin> graph = TitanFactory.open("/usr/iop/4.2.5.0-0000/titan/conf/titan-hbase-solr.properties")

gremlin> mgmt = graph.openManagement()

gremlin> desc = mgmt.getPropertyKey("desc2")

gremlin> mr = new MapReduceIndexManagement(graph)

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]:

hadoop gremlin> mgmt.set('hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: hadoop Display stack trace? [yN] n

gremlin> mgmt.set('titan.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: titan Display stack trace? [yN] n

gremlin> mgmt.set('mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapred Display stack trace? [yN] n gremlin>

gremlin> mgmt.set('mapreduce.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapreduce Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: mapred Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: hadoop Display stack trace? [yN] n gremlin>

1 ACCEPTED SOLUTION

avatar
Expert Contributor
@pbarna

I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved.

Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3.

My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect.

Example:

<property>
<name>mapreduce.local.map.tasks.maximum</name>
<value>10</value>
<final>true</final>
</property>

Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here.

Please feel free to reach out if you think this was useful.

Thanks @Nixon Rodrigues for letting me know about this question.

View solution in original post

2 REPLIES 2

avatar
Guru

avatar
Expert Contributor
@pbarna

I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved.

Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3.

My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect.

Example:

<property>
<name>mapreduce.local.map.tasks.maximum</name>
<value>10</value>
<final>true</final>
</property>

Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here.

Please feel free to reach out if you think this was useful.

Thanks @Nixon Rodrigues for letting me know about this question.