question Re: Solr installation in Archives of Support Questions (Read Only)

Solr installation

SK1 — Tue, 12 Jul 2016 19:24:27 GMT

Team,

I am new to Solr and want to install it in my cluster (5 nodes),before I go ahead I have got few questions.So can someone please help me on it.

1. Do I need to install solr on all nodes including master and workers ?

2. Can't we monitor it via Ambari ?

3. How we will configure Ranger Security on top of Solr ?

Note: I want to install solr in cloud mode(SolrCloud).

Re: Solr installation

lgeorge — Wed, 13 Jul 2016 01:12:00 GMT

Hi Saurabh, here is a partial response in case it's helpful: HDP Search (which includes Solr) should be deployed on all nodes that run HDFS. Ambari is not supported quite yet. The HDP Search Guide contains basic information and links to additional documentation.

Re: Solr installation

SK1 — Wed, 13 Jul 2016 19:04:26 GMT

Thanks a lot @lgeorge. So you mean to say If I have 5(2M+3W) nodes cluster then I have to install Solr on all 3 worker nodes only or I need to install on 1 master and all 3 worker nodes as a part of server & client.

Also I would be very thankful if can you please help me to get following answer as well.

1. If I will install HDP-Search and it has banana so whether banana will run as root user or banana user ?

2. Can we do ldap integration for both Solr and Banana UI ?

3. How many resources we need for HDP search like heap,RAM or CPU ?

Thanks in advance.

Re: Solr installation

lgeorge — Fri, 15 Jul 2016 22:25:12 GMT

Good questions. Not sure, but I'm checking. If I find answers I'll post them (or send the Solr expert this way 🙂

Re: Solr installation

PARTOMIA — Mon, 18 Jul 2016 02:46:45 GMT

@Saurabh Kumar

1. Solr does not follow Master - Slave model, rather its Leader - Follower model.

Each Solr node therefore will be used for Indexing/Query, in SolrCloud.

Considering that you have 5 nodes, the Solr Collection creation therefore, can be done with 2 Shards and RF (Replication Factor ) of 2. This will allow to use 4 nodes for Solr.

2. Each node which is supposed to be used for Solr, need to be installed with "lucidworks-hdpsearch".

3. Resource usage depends on the Size of Index ( present and estimated growth of index ). Refer following for further understanding on resource usage:

https://wiki.apache.org/solr/SolrPerformanceProblems

Re: Solr installation

SK1 — Wed, 20 Jul 2016 14:08:06 GMT

Thanks @Ravi

Re: Solr installation

SK1 — Wed, 20 Jul 2016 14:08:24 GMT

Thanks @lgeorge.

Re: Solr installation

SK1 — Wed, 20 Jul 2016 17:01:57 GMT

@Ravi Can you please help me how to setup buffer memory for my solr cluster. I am getting following error.

[solr@m1 solr]$ /opt/lucidworks-hdpsearch/solr/bin/solr create -c test -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n test -s 2 -rf 2

Connecting to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Uploading /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf for config test to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Creating new collection 'test' using command:

http://192.168.56.42:8983/solr/admin/collections?action=CREATE&name=test&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=test

{

"responseHeader":{

"status":0,

"QTime":4812},

"failure":{"":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.56.41:8983/solr: Error CREATEing SolrCore 'test_shard1_replica1': Unable to create core [test_shard1_replica1] Caused by: Direct buffer memory"},

"success":{"":{

"responseHeader":{

"status":0,

"QTime":4659},

"core":"test_shard2_replica1"}}}

Re: Solr installation

PARTOMIA — Thu, 21 Jul 2016 02:28:23 GMT

@Saurabh Kumar

The error which are you getting is :

"Unable to create core [test_shard1_replica1] Caused by: Direct buffer memory"} "

Looks to me that you have set up the Direct Memory ( to enable Block Cache ) as true in the "solrconfig.xml" file i.e.

<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>

From your "solrconfig.xml", I see the config as:

<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://m1.hdp22:8020/user/solr</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>

I will suggest to turn off the Direct Memory if you do not plan to use it for now and then try the creation of collection.

To disable it, edit the "solrconfig.xml" and looks for property - "solr.hdfs.blockcache.direct.memory.allocation".

Make the value of this property to "false" i.e.

<bool name="solr.hdfs.blockcache.direct.memory.allocation">false</bool>

The final "solrconfig.xml" will therefore look like :

                <directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">                  <str name="solr.hdfs.home">hdfs://m1.hdp22:8020/user/solr</str>
                <bool name="solr.hdfs.blockcache.enabled">true</bool>
                <int name="solr.hdfs.blockcache.slab.count">1</int>
                <bool name="solr.hdfs.blockcache.direct.memory.allocation">false</bool>
                <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
                <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
                <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
                <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
                <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
                <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
                </directoryFactory>

Re: Solr installation

SK1 — Thu, 21 Jul 2016 14:32:19 GMT

@Ravi: Thanks a lot, It helped me to avoid direct memory issue but now I encountered another issue, so can you please help me on this also.

[solr@m1 solr]$ /opt/lucidworks-hdpsearch/solr/bin/solr create -c test -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n test -s 2 -rf 2

Connecting to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Uploading /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf for config test to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Creating new collection 'test' using command:

http://192.168.56.41:8983/solr/admin/collections?action=CREATE&name=test&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=test

{

"responseHeader":{

"status":0,

"QTime":6299},

"success":{"":{

"responseHeader":{

"status":0,

"QTime":5221},

"core":"test_shard2_replica1"}}}

Re: Solr installation

SK1 — Thu, 21 Jul 2016 14:46:18 GMT

@Ravi

Hey Ravi, thanks I have solved it by changing value of SOLR_HEAP to 1024 MB in /opt/lucidworks-hdpsearch/solr/bin/solr.in.sh. Thanks once again for all your help.

SOLR_HEAP="1024m"

[solr@m1 solr]$ /opt/lucidworks-hdpsearch/solr/bin/solr create -c test -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n test -s 2 -rf 2

Connecting to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Uploading /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf for config test to ZooKeeper at m1.hdp22:2181,m2.hdp22:2181,w1.hdp22:2181

Creating new collection 'test' using command:

http://192.168.56.42:8983/solr/admin/collections?action=CREATE&name=test&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=test

{

"responseHeader":{

"status":0,

"QTime":8494},

"success":{"":{

"responseHeader":{

"status":0,

"QTime":8338},

"core":"test_shard1_replica1"}}}

Re: Solr installation

PARTOMIA — Thu, 21 Jul 2016 15:13:20 GMT

@Saurabh Kumar

You are welcome.

For the issue with Java heap space , its due to Java_Heap for Solr Process. By default Solr process is started with only 512MB. We can increase this by editing the Solr config files or via solr command line options as:

/opt/lucidworks-hdpsearch/solr/bin/solr -m 2g create -c test -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n test -s 2 -rf 2

This will resolve the Java heap space issue.

Re: Solr installation

SK1 — Fri, 22 Jul 2016 12:27:25 GMT

Thanks @Ravi. I have solved it by changing value of SOLR_HEAP to 1024 MB in /opt/lucidworks-hdpsearch/solr/bin/solr.in.sh. Thanks once again for all your help.

SOLR_HEAP="1024m"

[solr@m1 solr]$ /opt/lucidworks-hdpsearch/solr/bin/solr create -c test -d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n test -s 2 -rf 2

Re: Solr installation

josephmontenaro — Fri, 30 Dec 2016 22:48:28 GMT

Hi Saurabh,

I know this is an older question, but if you (or anyone else) is still looking to monitor Solr Cloud via Ambari, laying a custom service on top of your existing installation might be useful. The following will allow you to integrate Solr Cloud into Ambari, complete with alerts and the ability to start, stop, and monitor status.

This setup assumes an existing, standard Solr Cloud installation, with the Sold Cloud UI available on port 8983.

On the Ambari node, create /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/SOLR/package/scripts.

In /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/SOLR, create alerts.json and metainfo.xml, as follows (you can, of course, change the version to whatever version of Solr you have installed):

alerts.json

{
  "SOLR": {
    "service": [],
    "SOLR_CLOUD": [
      {
        "name" : "solr_cloud_ui",
        "label" : "Solr Cloud UI",
        "description" : "This host-level alert is triggered if the Solr Cloud Web UI is unreachable.",
        "interval" : 1,
        "scope" : "ANY",
        "source" : {
          "type" : "WEB",
          "uri" : {
            "http" : "http://0.0.0.0:8983",
            "connection_timeout" : 5.0
          },
          "reporting" : {
            "ok" : {
              "text" : "HTTP {0} response in {2:.3f}s"
            },
            "warning" : {
              "text" : "HTTP {0} response from {1} in {2:.3f}s ({3})"
            },
            "critical" : {
              "text" : "Connection failed to {1} ({3})"
            }
          }
        }
      }
    ]
  }
}

metainfo.xml

<?xml version="1.0"?>
<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <name>SOLR</name>
      <displayName>Solr</displayName>
      <comment>Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project.</comment>
      <version>5.2.1</version>
      <components>
        <component>
          <name>SOLR_CLOUD</name>
          <displayName>Solr Cloud Server</displayName>
          <category>MASTER</category>
          <cardinality>1+</cardinality>
          <commandScript>
            <script>scripts/solrcloud.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>600</timeout>
          </commandScript>
        </component>
      </components>
    </service>
  </services>
</metainfo>

In /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/SOLR/package/scripts, create params.py and solrcloud.py, as follows:

params.py

cloud_stop = ('/sbin/service', 'solr', 'stop')
cloud_start = ('/sbin/service', 'solr', 'start')
cloud_pid_file = '/opt/lucidworks-hdpsearch/solr/bin/solr-8983.pid'

solrcloud.py

from resource_management import *
from resource_management.core.resources.system import Execute

class Master(Script):
  def install(self, env):
    print 'Installing Solr Cloud';

  def stop(self, env):
    import params
    env.set_params(params)

    Execute((params.cloud_stop), sudo=True)

  def start(self, env):
    import params
    env.set_params(params)

    Execute((params.cloud_start), sudo=True)

  def status(self, env):
    import params
    env.set_params(params)

    from resource_management.libraries.functions import check_process_status

    check_process_status(params.cloud_pid_file)

  def configure(self, env):
    print 'Configuring Solr Cloud';

if __name__ == "__main__":
  Master().execute()

At this point, after restarting Ambari, you will be able to "install" Solr Cloud via the Ambari Add Service wizard, specifying a Solr Cloud Server on whichever hosts Solr is already installed. As you might note from solrcloud.py, the installation doesn't do anything other than configure Ambari to be aware that the components exist on the hosts.

Once the installation is complete, Solr will be listed as an Ambari Service, with each Solr Cloud server listed as an individual Master component.

Hope this helps.

Joe