Batch Indexing Using MapReduce to Solr

[#1]

Hi,

When I try to example of Batch Indexing into Online Solr Servers Using GoLive, i cannot index to solr because of this warning in solr logs =>

WARN

HdfsLocalityReporter

Could not retrieve locality information for hdfs://...:8020/user/emre/outdir/results/part-00001/data...

Although file is exist in path hdfs://...:8020/user/emre/outdir/results/part-00001/data/index, i got this exception.

Emre,

Thanks

[#2]

Which version of Cloudera Search are you using?

That warning happens when trying to load some metrics on the data in HDFS. Are there any other warnings or errors in the logs that you can find?

This warning is probably safe to ignore in your case.

[#3]

I'm using search-1.0.0+cdh5.4.5+0 version.

I got another exception java.io.FileNotFoundException: File hdfs://...:8020/user/emre/outdir/results/part-00000/data/index does not exist. 

But, related file is exist.

Thanks,

Emre

[#4]

Hi Emre,

Thanks for updating me on your version. That will help me find the cause of the problem.

Can you provide more context from your logs? This second exception looks like the same thing, but for a different task output.

Can you be more specific about what kind of failure you see?

[#5]

I cannot see another error in solr log.  

MapReduceIndexerTool is working succesfully. Details => 

497 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Indexing 2 files using 2 real mappers into 2 reducers
20730 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Done. Indexing 2 files using 2 real mappers into 2 reducers took 6.7444393E9 secs
20741 [main] INFO org.apache.solr.hadoop.GoLive - Live merging of output shards into Solr cluster...
20744 [pool-4-thread-2] INFO org.apache.solr.hadoop.GoLive - Live merge hdfs://...:8020/user/emre/outdir/results/part-00001 into http://...:8983/solr
20744 [pool-4-thread-1] INFO org.apache.solr.hadoop.GoLive - Live merge hdfs://...:8020/user/emre/outdir/results/part-00000 into http://...:8983/solr
20885 [main] INFO org.apache.solr.hadoop.GoLive - Committing live merge...
20892 [main] INFO org.apache.solr.common.cloud.SolrZkClient - Using default ZkCredentialsProvider
20893 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Waiting for client to connect to ZooKeeper
20897 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager - Watcher org.apache.solr.common.cloud.ConnectionManager@61d818f6 name:ZooKeeperConnection Watcher:...:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
20897 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Client is connected to ZooKeeper
20897 [main] INFO org.apache.solr.common.cloud.SolrZkClient - Using default ZkACLProvider
20898 [main] INFO org.apache.solr.common.cloud.ZkStateReader - Updating cluster state from ZooKeeper...
20922 [main] INFO org.apache.solr.hadoop.GoLive - Done committing live merge
20922 [main] INFO org.apache.solr.hadoop.GoLive - Live merging of index shards into Solr cluster took 6.0152416E7 secs
20922 [main] INFO org.apache.solr.hadoop.GoLive - Live merging completed successfully
20922 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Succeeded with job: jobName: org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, jobId: job_1447398200936_0024
20922 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Success. Done. Program took 6.9784602E9 secs. Goodbye.

But, i cannot see any doc in solr index.  When run solr query that t...:8983/solr/collection1/select?q=*%3A*&wt=json&indent=true , number of found is 0.

[#6 · Accepted solution]

I mistaken morphline configuration. Because of this mistake, docs could not indexed in solr. 

Thanks for responces,

Emre