Thanks @Jonas Straub, @Michael Young
I have been distracted for a few days but now I'm back to this issue again.
I tried playing around with the sleep time in the stop process, but it didn't seem to have any impact.
Now I'm trying to understand the maxShardsPerNode parameter. I have it set to 1 for my collection:
http://xxxxxx:34010/solr/admin/collections?action=CREATE&name=aircargo_cloud&numShards=4&replication...
As I mentioned earlier, if I create this collection when my 4 nodes are active, all is well. Each node gets one shard and everything works very well. Remember, I am using HDFS for my index storage.
If I shut down the 4 nodes and then bring the first one back up, it grabs all 4 shards which seems contrary to the NumShardsPerNode setting. I expected the first node to grab one shard and for the other 3 to remain unassigned. (At this stage, with just a single node running, the collection is accessible and queries run successfully.)
Because the first node acquires the 4 shards, it is not surprising that when I bring up the second node that it recognises that it's shard is already locked.
Once the second node comes up (and reports the locking error), the collection become unusable. An attempted query, returns:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/aircargo_cloud/select. Reason:
<pre> Not Found</pre></p>
</body>
</html>
Am I misunderstanding the purpose/operation of the maxShardsPerNode parameter?
Am I confusing things by trying to run all 4 nodes on the same server? I have a chunky server available (it has 40 cores and 256GB of memory). It has a very fast connection to the Hadoop cluster where I'm storing the Solr indexes. So it seems like a reasonable configuration from a resource availability viewpoint.
Each node has the same 'HOST_NAME' (as defined in solr.in.sh). Is this confusing things?
When I bring up just the first node (and queries work OK), the state.json file (available in zookeeper) looks as follows (with some obfuscation). You can see that the node_name is the same for all 4 shards (even though, right at the bottom we see: "maxShardsPerNode":"1" )
{"aircargo_cloud":{
"replicationFactor":"1",
"shards":
{
"shard1":{
"range":"80000000-bfffffff",
"state":"active",
"replicas":{"core_node1":{
"core":"aircargo_cloud_shard1_replica1",
"dataDir":"hdfs://HDP/solr/aircargo_cloud/core_node1/data/",
"base_url":"http://XXXXXX.YYYYYY.ZZZZZZ:34010/solr",
"node_name":"XXXXXX.YYYYYY.ZZZZZZ:34010_solr",
"state":"active",
"ulogDir":"hdfs://HDP/solr/aircargo_cloud/core_node1/data/tlog",
"leader":"true"}}},
"shard2":{
"range":"c0000000-ffffffff",
"state":"active",
"replicas":{"core_node3":{
"core":"aircargo_cloud_shard2_replica1",
"dataDir":"hdfs://HDP/solr/aircargo_cloud/core_node3/data/",
"base_url":"http://XXXXXX.YYYYYY.ZZZZZZ:34010/solr",
"node_name":"XXXXXX.YYYYYY.ZZZZZZ:34010_solr",
"state":"active",
"ulogDir":"hdfs://HDP/solr/aircargo_cloud/core_node3/data/tlog",
"leader":"true"}}},
"shard3":{
"range":"0-3fffffff",
"state":"active",
"replicas":{"core_node4":{
"core":"aircargo_cloud_shard3_replica1",
"dataDir":"hdfs://HDP/solr/aircargo_cloud/core_node4/data/",
"base_url":"http://XXXXXX.YYYYYY.ZZZZZZ:34010/solr",
"node_name":"XXXXXX.YYYYYY.ZZZZZZ:34010_solr",
"state":"active",
"ulogDir":"hdfs://HDP/solr/aircargo_cloud/core_node4/data/tlog",
"leader":"true"}}},
"shard4":{
"range":"40000000-7fffffff",
"state":"active",
"replicas":{"core_node2":{
"core":"aircargo_cloud_shard4_replica1",
"dataDir":"hdfs://HDP/solr/aircargo_cloud/core_node2/data/",
"base_url":"http://XXXXXX.YYYYYY.ZZZZZZ:34010/solr",
"node_name":"XXXXXX.YYYYYY.ZZZZZZ:34010_solr",
"state":"active",
"ulogDir":"hdfs://HDP/solr/aircargo_cloud/core_node2/data/tlog",
"leader":"true"}}}
},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false"}}
Regards
Tony
