Solr core/shard backup and restore function ONLY became available after version 5.2 . See JIRA: https://issues.apache.org/jira/browse/SOLR-6637 To manually backup and restore shards, please follow the steps : Assumes a SolrCloud cluster has 4 cores and 2 node, which 2 cores on each node. (2 shards, with replicationFactor=2) Back-up each node: http://server1:8983/solr/collection/replication?command=backup&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=backup&location=location2&name=backup2 Create new collection with 2 shards and replicationFactor=2 http://server:8983/solr/admin/collections?action=CREATE&name=newCollection&numShards=2&replicationFactor=2&createNodeSet=server1:8983_solr,server2:8983_solr Restore from each backup: http://server1:8983/solr/newCollection/replication?command=restore&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=restore&location=location2&name=backup2 Add replicas to each shard http://server:8983/solr/admin/collections?action=ADDREPLICA&collection=newCollection&shard=shard2&node=server1:8983_solr http://server:8983/solr/admin/collections?action=ADDREPLICA&collection=newCollection&shard=shard1&node=server2:8983_solr In the API commands where we just have "server" without 1 or 2, it means you can issue the command from either server. ... View more

Following is a guide to split shards and add replicas for collection "ranger_audits" on SolrCloud , in case the initial ranger_audits is created with only 1 shard 1 replica : 1.Initial ranger_audits was created with only 1 shard and 1 replica and user would like to split the shard to balance load and add more replicas for back up. 2. In the web browser, enter URL to split the shard : http://172.25.17.51:6083/solr/admin/collections?action=SPLITSHARD&collection=ranger_audits&shard=shard1 , and user shall get a page like which means the splitting operation is successful. And in the Solr Admin UI, user can see there are two additional shards : Notice, the original shard1 still exists. This is because splitting a shard will take an existing shard and break it into two pieces which are written to disk as two (new) shards. The original shard will continue to contain the same data as-is but it will start re-routing requests to the new shards. The new shards will have as many replicas as the original shard. A soft commit is automatically issued after splitting a shard so that documents are made visible on sub-shards. An explicit commit (hard or soft) is not necessary after a split operation because the index is automatically persisted to disk during the split operation. This command allows for seamless splitting and requires no downtime. A shard being split will continue to accept query and indexing requests and will automatically start routing them to the new shards once this operation is complete. This command can only be used for SolrCloud collections created with "numShards" parameter, meaning collections which rely on Solr's hash-based routing mechanism. The split is performed by dividing the original shard's hash range into two equal partitions and dividing up the documents in the original shard according to the new sub-ranges. 3. Create replicas on a different solr node with the SolrCloud for each new shards : http://172.25.17.51:6083/solr/admin/collections?action=ADDREPLICA&collection=ranger_audits&shard=shard1_0&node=172.25.17.45:6083_solr http://172.25.17.51:6083/solr/admin/collections?action=ADDREPLICA&collection=ranger_audits&shard=shard1_1&node=172.25.17.45:6083_solr Notice the original solr node still has "Leader" role for both new shards. This is not all that important to worry about. The additional duties of a leader are pretty minimal. And the leaders will shift around anyway as you restart servers etc. ... View more

Cluster setup : Client hostnames : mf-amos110.novalocal , and mf-amos111.novalocal Server hostnames : node1.openstacklocal , node2.openstacklocal and node3.openstacklocal step 1. create Company CA "rootCA.key" , on host mf-amos111.novalocal : [root@mf-amos111 centos]# openssl genrsa -aes256 -out rootCA.key 4096 Step 2 . create a pem file "rootCA.pem", which will be used to sign certificates sent by other hosts : [root@mf-amos111 centos]# openssl req -x509 -new -key rootCA.key -days 1095 -out rootCA.pem Step 3. Create a truststore file "all-truststore.jks" for all hosts, and copy this keystore to /etc/hadoop/conf dir on each host. [root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore all-truststore.jks -alias CARoot -import -file rootCA.pem [root@mf-amos111 centos]# cp all-truststore.jks /etc/hadoop/conf [root@mf-amos111 conf]# scp all-truststore.jks root@mf-amos110.novalocal:/etc/hadoop/conf all-truststore.jks 100% 1506 1.5KB/s 00:00 [root@mf-amos111 conf]# scp all-truststore.jks root@node1:/etc/hadoop/conf all-truststore.jks 100% 1506 1.5KB/s 00:00 [root@mf-amos111 conf]# scp all-truststore.jks root@node2:/etc/hadoop/conf all-truststore.jks 100% 1506 1.5KB/s 00:00 [root@mf-amos111 conf]# scp all-truststore.jks root@node3:/etc/hadoop/conf all-truststore.jks 100% 1506 1.5KB/s 00:00 [root@mf-amos111 conf]# Step 4. Create private keystore on each host in both clusters : [root@node1 centos]# keytool -genkey -alias node1 -keyalg RSA -keystore node1-keystore.jks -keysize 2048 .... [root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -genkey -alias amos111 -keyalg RSA -keystore amos111-keystore.jks -keysize 2048 NOTE : make sure the answer for "What is your first and last name?" is each of your host's FQDN !! Step 5. Generate a request to sign file ".csr" on each host [root@node1 centos]# keytool -certreq -alias node1 -keystore node1-keystore.jks -file node1-keystore.csr .... [root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -certreq -alias amos111 -keystore amos111-keystore.jks -file amos111-keystore.csr Step 6: scp the csr file to mf-amos111.novalocal , to get them signed by rootCA [root@node1 centos]# scp node1-keystore.csr root@mf-amos111.novalocal:/home/centos ... [root@mf-amos110 centos]# scp amos110-keystore.csr root@mf-amos111.novalocal:/home/centos Step 7 : sign the csr files on mf-amos111.novalocal , and get a crt file for each host [root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos111-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos111-keystore.crt -days 730 Signature ok subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos111.novalocal Getting CA Private Key Enter pass phrase for rootCA.key: [root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos110-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos110-keystore.crt -days 730 Signature ok subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos110.novalocal Getting CA Private Key Enter pass phrase for rootCA.key: [root@mf-amos111 centos]# openssl x509 -sha256 -req -in node1-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node1-keystore.crt -days 730 Signature ok subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node1.openstacklocal Getting CA Private Key Enter pass phrase for rootCA.key: [root@mf-amos111 centos]# openssl x509 -sha256 -req -in node2-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node2-keystore.crt -days 730 Signature ok subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node2.openstacklocal Getting CA Private Key Enter pass phrase for rootCA.key: [root@mf-amos111 centos]# openssl x509 -sha256 -req -in node3-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node3-keystore.crt -days 730 Signature ok subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node3.openstacklocal Getting CA Private Key Enter pass phrase for rootCA.key: [root@mf-amos111 centos]# Step 8 : Copy the crt file, and the rootCA.pem file to each host in both clusters : [root@mf-amos111 centos]# scp rootCA.pem root@mf-amos110.novalocal:/home/centos rootCA.pem 100% 2009 2.0KB/s 00:00 [root@mf-amos111 centos]# scp rootCA.pem root@node1:/home/centos rootCA.pem 100% 2009 2.0KB/s 00:00 [root@mf-amos111 centos]# scp rootCA.pem root@node2:/home/centos rootCA.pem 100% 2009 2.0KB/s 00:00 [root@mf-amos111 centos]# scp rootCA.pem root@node3:/home/centos rootCA.pem 100% 2009 2.0KB/s 00:00 [root@mf-amos111 centos]# scp amos110-keystore.crt root@mf-amos110.novalocal:/home/centos amos110-keystore.crt 100% 1566 1.5KB/s 00:00 [root@mf-amos111 centos]# scp node1-keystore.crt root@node1:/home/centos node1-keystore.crt 100% 1566 1.5KB/s 00:00 [root@mf-amos111 centos]# scp node2-keystore.crt root@node2:/home/centos node2-keystore.crt 100% 1566 1.5KB/s 00:00 [root@mf-amos111 centos]# scp node3-keystore.crt root@node3:/home/centos node3-keystore.crt 100% 1566 1.5KB/s 00:00 Step 9 : Import the crt file, and the rootCA.pem to the keystore files on each host [root@node1 centos]# keytool -keystore node1-keystore.jks -alias rootCA -import -file rootCA.pem Enter keystore password: Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US Serial number: fbe9c20f767c716d Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020 Certificate fingerprints: MD5: 62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30 SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 78 8D 63 37 DF 19 BE C5 BC 42 36 09 57 FD 70 FF x.c7.....B6.W.p. 0010: C4 AB 76 D2 ..v. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] #3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 78 8D 63 37 DF 19 BE C5 BC 42 36 09 57 FD 70 FF x.c7.....B6.W.p. 0010: C4 AB 76 D2 ..v. ] ] Trust this certificate? [no]: yes Certificate was added to keystore [root@node1 centos]# keytool -keystore node1-keystore.jks -alias node1 -import -file node1-keystore.crt Enter keystore password: Certificate reply was installed in keystore ..... ..... [root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias rootCA -import -file rootCA.pem Enter keystore password: Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US Serial number: fbe9c20f767c716d Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020 Certificate fingerprints: MD5: 62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30 SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E Signature algorithm name: SHA1withRSA Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 78 8D 63 37 DF 19 BE C5 BC 42 36 09 57 FD 70 FF x.c7.....B6.W.p. 0010: C4 AB 76 D2 ..v. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] #3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 78 8D 63 37 DF 19 BE C5 BC 42 36 09 57 FD 70 FF x.c7.....B6.W.p. 0010: C4 AB 76 D2 ..v. ] ] Trust this certificate? [no]: yes Certificate was added to keystore [root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias amos111 -import -file amos111-keystore.crt Enter keystore password: Certificate reply was installed in keystore Step 10 : Copy the latest keystore files to /etc/hadoop/conf dir as well, [root@node1 centos]# cp /home/centos/node1-keystore.jks /etc/hadoop/conf/all-keystore.jks [root@node2 centos]# cp /home/centos/node2-keystore.jks /etc/hadoop/conf/all-keystore.jks [root@node3 centos]# cp /home/centos/node3-keystore.jks /etc/hadoop/conf/all-keystore.jks [root@mf-amos110 centos]# cp /home/centos/amos110-keystore.jks /etc/hadoop/conf/all-keystore.jks [root@mf-amos111 conf]# cp /home/centos/amos111-keystore.jks /etc/hadoop/conf/all-keystore.jks Step 11 : configure following properties in both Ambari -> hdfs config for both clusters. Advanced ssl-client -> ssl.client.keystore.location -> /etc/hadoop/conf/all-keystore.jks Advanced ssl-client -> ssl.client.truststore.location -> /etc/hadoop/conf/all-truststore.jks change the passwords accordingly Advanced ssl-server -> ssl.server.truststore.location -> /etc/hadoop/conf/all-truststore.jks Advanced ssl-server -> ssl.server.keystore.location -> /etc/hadoop/conf/all-keystore.jks change the passwords accordingly Advanced hdfs-site -> dfs.http.policy -> HTTP_AND_HTTPS Step 12 : restart both HDFS Step 13 : test [hdfs@mf-amos111 ~]$ hadoop distcp swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917 swebhdfs://node1.openstacklocal/tmp/ 17/03/09 23:23:31 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917], targetPath=swebhdfs://node1.openstacklocal/tmp, targetPathExists=true, filtersFile='null'} 17/03/09 23:23:31 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/ 17/03/09 23:23:31 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050 17/03/09 23:23:31 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200 17/03/09 23:23:33 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0 17/03/09 23:23:33 INFO tools.SimpleCopyListing: Build file listing completed. 17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1 17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1 17/03/09 23:23:33 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/ 17/03/09 23:23:33 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050 17/03/09 23:23:33 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200 17/03/09 23:23:34 INFO mapreduce.JobSubmitter: number of splits:1 17/03/09 23:23:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489078447425_0004 17/03/09 23:23:34 INFO impl.YarnClientImpl: Submitted application application_1489078447425_0004 17/03/09 23:23:34 INFO mapreduce.Job: The url to track the job: http://mf-amos111.novalocal:8088/proxy/application_1489078447425_0004/ 17/03/09 23:23:34 INFO tools.DistCp: DistCp job-id: job_1489078447425_0004 17/03/09 23:23:34 INFO mapreduce.Job: Running job: job_1489078447425_0004 17/03/09 23:23:42 INFO mapreduce.Job: Job job_1489078447425_0004 running in uber mode : false 17/03/09 23:23:42 INFO mapreduce.Job: map 0% reduce 0% 17/03/09 23:23:50 INFO mapreduce.Job: map 100% reduce 0% 17/03/09 23:23:50 INFO mapreduce.Job: Job job_1489078447425_0004 completed successfully 17/03/09 23:23:50 INFO mapreduce.Job: Counters: 38 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=143285 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=375 HDFS: Number of bytes written=0 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 SWEBHDFS: Number of bytes read=1309 SWEBHDFS: Number of bytes written=1309 SWEBHDFS: Number of read operations=11 SWEBHDFS: Number of large read operations=0 SWEBHDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=5098 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5098 Total vcore-milliseconds taken by all map tasks=5098 Total megabyte-milliseconds taken by all map tasks=5220352 Map-Reduce Framework Map input records=1 Map output records=0 Input split bytes=115 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=91 CPU time spent (ms)=3190 Physical memory (bytes) snapshot=252891136 Virtual memory (bytes) snapshot=2787827712 Total committed heap usage (bytes)=113770496 File Input Format Counters Bytes Read=260 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=1309 BYTESEXPECTED=1309 COPY=1 [hdfs@mf-amos111 ~]$ ... View more