Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Installing Solr on yarn using Slider

avatar
Contributor

Hi,

I am trying to run solr on yarn using the link lucidworksSolrSlider, apart from taking help from slider.incubator.apache.org/docs/getting_started.html

Here is my folder structure:

[solrs@ip-10-0-0-217 solr-slider]$ ls -lrt
total 131744
-rw-rw-r--. 1 solrs solrs      3182 Dec 10 01:17 README.md
drwxrwxr-x. 4 solrs solrs        32 Dec 10 01:17 package
-rw-rw-r--. 1 solrs solrs      2089 Dec 10 01:17 metainfo.xml
-rw-rw-r--. 1 solrs solrs     11358 Dec 10 01:17 LICENSE
-rw-rw-r--. 1 solrs solrs 134874517 Dec 10 01:37 solr-on-yarn.zip
-rw-rw-r--. 1 solrs solrs       277 Dec 10 01:49 resources-default.json
-rw-rw-r--. 1 solrs solrs      1355 Dec 10 15:33 appConfig-default.json

appConfig-default.json:

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "application.def": "/user/solrs/.slider/package/solryarn/solr-on-yarn.zip",
    "java_home": "/usr/jdk64/jdk1.8.0_40",
    "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT",
    "site.global.zk_host": "localhost:2181",
    "site.global.solr_host": "${SOLR_HOST}",
    "site.global.listen_port": "${SOLR.ALLOCATED_PORT}",
    "site.global.xmx_val": "1g",
    "site.global.xms_val": "1g",
    "site.global.gc_tune": "-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewG$
    "site.global.zk_timeout": "15000",
    "site.global.server_module": "--module=http",
    "site.global.stop_key": "solrrocks",
    "site.global.solr_opts": ""
  },
  "components": {
    "slider-appmaster": {
      "jvm.heapsize": "512M"
    },
    "SOLR": {
    }
  }
}

resources-default.json:

{
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : {
  },
  "global" : {
  },
  "components": {
    "slider-appmaster": {
    },
    "SOLR": {
      "yarn.role.priority": "1",
      "yarn.component.instances": "3",
      "yarn.memory": "1024"
    }
  }
}

Could you please suggest me what will be the value of below parameters in appConfig-default.json file:

"site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT",
"site.global.solr_host": "${SOLR_HOST}",
"site.global.listen_port": "${SOLR.ALLOCATED_PORT}",

Basically where should I find "/app/install/solr-5.2.0-SNAPSHOT"??

My Environment: HDP 2.3, Slider Core-0.80.0.2.3.2.0-2950

Thanks, hoping a quick reply.

1 ACCEPTED SOLUTION

avatar
Expert Contributor

The only part of "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT" that you should change is the solr-5.2.0-SNAPSHOT. You should make this match the version of the Solr tarball you downloaded. (You can check the version by running "tar tf solr.tgz").

You probably also want to change "site.global.zk_host": "localhost:2181" to "site.global.zk_host": "${ZK_HOST}", which will configure Solr to use the same ZooKeeper instance Slider is using.

I think you can leave ${SOLR_HOST} as is, but I am not completely sure of the purpose of that parameter.

View solution in original post

12 REPLIES 12

avatar
Expert Contributor

I guess this is a design limitation of the LucidWorks solr app package. One thing you'd have to do to avoid losing data on restart is to configure Solr to store its data in HDFS. You might be able to do this by setting the solr_opts parameter in the appConfig:

-Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=none -Dsolr.hdfs.confdir=/etc/hadoop/conf -Dsolr.hdfs.home=${DEFAULT_DATA_DIR}

You would also need to configure automatic failover when creating the collection:

curl "http://server:port/solr/admin/collections?action=CREATE&name=collection_name&numShards=2&autoAddReplicas=true"

avatar
Expert Contributor

Slider has to run the Solr servers on random ports because it can't control where they are started (more than one might be started on the same host). The next release of Slider will have an anti-affinity container placement option, so that might help. You should be able to connect to each of the Solr instances that Slider creates.

avatar
Master Mentor

@Rakesh Gupta excellent question, are you still having issues with this? Can you accept best answer or provide your own solution?