Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can yarn queue be specified for Titan mapreduce indexing jobs?

Solved Go to solution

How can yarn queue be specified for Titan mapreduce indexing jobs?

Contributor

whatever I tried, all yarn applications ended up in the default queue

these are the things I tried:

1) setting property in the titan-hbase-solr.properties (none of the following worked)

mapred.job.queue.name=myqueue

mapreduce.job.queue.name=myqueue

mapred.mapreduce.job.queue.name=myqueue

2) setting property in the gremlin shell

gremlin> graph = TitanFactory.open("/usr/iop/4.2.5.0-0000/titan/conf/titan-hbase-solr.properties")

gremlin> mgmt = graph.openManagement()

gremlin> desc = mgmt.getPropertyKey("desc2")

gremlin> mr = new MapReduceIndexManagement(graph)

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]:

hadoop gremlin> mgmt.set('hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: hadoop Display stack trace? [yN] n

gremlin> mgmt.set('titan.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: titan Display stack trace? [yN] n

gremlin> mgmt.set('mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapred Display stack trace? [yN] n gremlin>

gremlin> mgmt.set('mapreduce.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root]: mapreduce Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: mapred Display stack trace? [yN] n

gremlin> mgmt.set('gremlin.hadoop.mapred.job.queue.name', 'myqueue')

Unknown configuration element in namespace [root.gremlin]: hadoop Display stack trace? [yN] n gremlin>

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How can yarn queue be specified for Titan mapreduce indexing jobs?

Expert Contributor
@pbarna

I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved.

Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3.

My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect.

Example:

<property>
<name>mapreduce.local.map.tasks.maximum</name>
<value>10</value>
<final>true</final>
</property>

Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here.

Please feel free to reach out if you think this was useful.

Thanks @Nixon Rodrigues for letting me know about this question.

2 REPLIES 2
Highlighted

Re: How can yarn queue be specified for Titan mapreduce indexing jobs?

Guru

Re: How can yarn queue be specified for Titan mapreduce indexing jobs?

Expert Contributor
@pbarna

I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved.

Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3.

My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect.

Example:

<property>
<name>mapreduce.local.map.tasks.maximum</name>
<value>10</value>
<final>true</final>
</property>

Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here.

Please feel free to reach out if you think this was useful.

Thanks @Nixon Rodrigues for letting me know about this question.