I wonder how's the replication work on solr? One of the core_node data directory is crashed, will solr automatically recover the lost data? If not, how to do it manually? The replication factor is set to 2 for this collection
Now I am still able to query without error messages but sometime it returns the correct total number of documents and sometime it show roughly 50% of it.
Awesome. If it continues to happen you'll need to figure out why you're getting OOM (assuming that was what was happening). Intermittent exceptions are often a symptom of OOM exceptions. Solr loves memory but there are a lot of factors that can contribute to it. Sometimes giving the JVM more memory is the solution, but not always. Good luck.