Ho to run solr index job in a specific yarn queue

[#1]

Team,

When i run solr index job then it run in default queue so is there any property or solution where I can change it to some other queue.

[root@m2 solr]# hadoop jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

[#2]

@Saurabh Kumar

I think you can use an option "queue" to specify the queue. Please see line 133 of SolrClient.java here.

https://github.com/lucidworks/yarn-proto/blob/master/src/main/java/org/apache/solr/cloud/yarn/SolrCl...

Also, follow this instruction if you run into any issue.

https://lucidworks.com/blog/2014/12/01/solr-yarn/

[#3]

@mqureshi: I tried but getting below error.

[solr@m1 solr]$ hadoop jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar -queue=ado com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

WARNING: Use "yarn jar" to launch YARN applications.

Exception in thread "main" java.lang.ClassNotFoundException: -queue=ado

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:270)

at org.apache.hadoop.util.RunJar.run(RunJar.java:214)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

I also tested below one but getting same error.

[solr@m1 solr]$ yarn jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar -queue=ado com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

Exception in thread "main" java.lang.ClassNotFoundException: -queue=ado

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:270)

at org.apache.hadoop.util.RunJar.run(RunJar.java:214)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

[#4 · Accepted solution]

@Saurabh Kumar

You might be following a different set of instructions. The one i am looking at uses solr-yarn.jar and here is the format

hadoop jar solr-yarn.jar org.apache.solr.cloud.yarn.SolrClient

-nodes=2

-zkHost=localhost:2181

-solr=hdfs://localhost:9000/solr/solr.tgz

-jar=hdfs://localhost:9000/solr/solr-yarn.jar

-memory 512

-hdfs_home=hdfs://localhost:9000/solr/index_data

This is where I think you should be able to add "-queue=<queue name>"

did you try following the instructions on this link? the class name which supports "queue" is not in the command you are running. Is it possible for you to try the following?

https://lucidworks.com/blog/2014/12/01/solr-yarn/

[#5]

Thanks @mqureshi. It is working now when I changed my jar to solr-yarn.jar. Actually earlier I was using separate jar program to create solr index. As Kuldeep mentioned I tried that property and it worked for my old program as well.

[#6]

@Saurabh Kumar

If your command is going to launch a Yarn job then how about adding -Dmapreduce.job.queuename=<queue-name> ?

[#7]

Thanks a lot @Kuldeep Kulkarni it is working for this jar.