We have an interesting situation with the MapReduceBatchIndexer tool where sometimes while the job finishes successfully, the indexes are not actually loaded into Solr via the live merge. The logs are too verbose to stick in this thread, but at the end the job says this. If we run it an un-determinate number of times more, then it will eventually work.
82774 [pool-4-thread-1] INFO org.apache.solr.hadoop.GoLive - Live merge hdfs://nameservice1/tmp/solredh_admin_user/results/part-00000 into http://mapls188.bsci.bossci.com:8983/solr ... 83073 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Succeeded with job: jobName: org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, jobId: job_1470234528819_0230 83073 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Success. Done. Program took 83.07273 secs. Goodbye.
Here is a snapshot of the current run. You can see that the Solr results are in our temp location.
We have enabled both DEBUG on the MapReduceBatchIndexer and the Solr server and have compared successful runs with none-successful runs without any luck identifying why sporadically this works and doesn't work.