Recover failed replica shard

[#1]

 we have 6 node cluster and solr installed on 4 machines. One of the collection size is 2TB data. Recently we upgraded cluster from 5.4.8 to 5.9.2. After the upgradation  the solr collection shards are in recovery state and 2 leader shards are recovered and active. but 2 replica shards got recovery failed after one day in recovery state.

How can recover the two replica shards from failed state?

Thanks in advance,

Ganesh

[#2]

Hi did you try to restart solr?

Also check solr log file. it hsould give you some clue

[#3]

Since it is production cluster can't do this operation. last time when we start it. It took almost 20 hours the collection become normal. We have around 2.9 billion records, which size is around 8 TB of data.