question Ho to run solr index job in a specific yarn queue in Archives of Support Questions (Read Only)

Ho to run solr index job in a specific yarn queue

SK1 — Tue, 02 Aug 2016 20:14:02 GMT

Team,

When i run solr index job then it run in default queue so is there any property or solution where I can change it to some other queue.

[root@m2 solr]# hadoop jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

Re: Ho to run solr index job in a specific yarn queue

mqureshi — Tue, 02 Aug 2016 22:52:10 GMT

@Saurabh Kumar

I think you can use an option "queue" to specify the queue. Please see line 133 of SolrClient.java here.

https://github.com/lucidworks/yarn-proto/blob/master/src/main/java/org/apache/solr/cloud/yarn/SolrClient.java

Also, follow this instruction if you run into any issue.

https://lucidworks.com/blog/2014/12/01/solr-yarn/

Re: Ho to run solr index job in a specific yarn queue

SK1 — Tue, 02 Aug 2016 23:58:04 GMT

@mqureshi: I tried but getting below error.

[solr@m1 solr]$ hadoop jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar -queue=ado com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

WARNING: Use "yarn jar" to launch YARN applications.

Exception in thread "main" java.lang.ClassNotFoundException: -queue=ado

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:270)

at org.apache.hadoop.util.RunJar.run(RunJar.java:214)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

I also tested below one but getting same error.

[solr@m1 solr]$ yarn jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar -queue=ado com.lucidworks.hadoop.ingest.IngestJob -DcsvFieldMapping=0=id,1=cat,2=name,3=price,4=instock,5=author -DcsvFirstLineComment -DidField=id -DcsvDelimiter="," -Dlww.commit.on.close=true -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c test -i csv/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -zk m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181/solr

Exception in thread "main" java.lang.ClassNotFoundException: -queue=ado

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:270)

at org.apache.hadoop.util.RunJar.run(RunJar.java:214)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Re: Ho to run solr index job in a specific yarn queue

mqureshi — Wed, 03 Aug 2016 01:37:47 GMT

@Saurabh Kumar

You might be following a different set of instructions. The one i am looking at uses solr-yarn.jar and here is the format

hadoop jar solr-yarn.jar org.apache.solr.cloud.yarn.SolrClient

-nodes=2

-zkHost=localhost:2181

-solr=hdfs://localhost:9000/solr/solr.tgz

-jar=hdfs://localhost:9000/solr/solr-yarn.jar

-memory 512

-hdfs_home=hdfs://localhost:9000/solr/index_data

This is where I think you should be able to add "-queue=<queue name>"

did you try following the instructions on this link? the class name which supports "queue" is not in the command you are running. Is it possible for you to try the following?

https://lucidworks.com/blog/2014/12/01/solr-yarn/

Re: Ho to run solr index job in a specific yarn queue

KuldeepK — Wed, 03 Aug 2016 09:28:01 GMT

@Saurabh Kumar

If your command is going to launch a Yarn job then how about adding -Dmapreduce.job.queuename=<queue-name> ?

Re: Ho to run solr index job in a specific yarn queue

SK1 — Wed, 03 Aug 2016 13:21:40 GMT

Thanks a lot @Kuldeep Kulkarni it is working for this jar.

Re: Ho to run solr index job in a specific yarn queue

SK1 — Wed, 03 Aug 2016 13:23:39 GMT

Thanks @mqureshi. It is working now when I changed my jar to solr-yarn.jar. Actually earlier I was using separate jar program to create solr index. As Kuldeep mentioned I tried that property and it worked for my old program as well.