we have 6 node cluster and solr installed on 4 machines. One of the collection size is 2TB data. Recently we upgraded cluster from 5.4.8 to 5.9.2. After the upgradation the solr collection shards are in recovery state and 2 leader shards are recovered and active. but 2 replica shards got recovery failed after one day in recovery state.
How can recover the two replica shards from failed state?
Since it is production cluster can't do this operation. last time when we start it. It took almost 20 hours the collection become normal. We have around 2.9 billion records, which size is around 8 TB of data.