question Re: Batch Indexing Using MapReduce to Solr in Support Questions

Batch Indexing Using MapReduce to Solr

heerkek — Fri, 16 Sep 2022 09:49:15 GMT

Hi,

When I try to example of Batch Indexing into Online Solr Servers Using GoLive, i cannot index to solr because of this warning in solr logs =>

WARN

HdfsLocalityReporter

Could not retrieve locality information for hdfs://...:8020/user/emre/outdir/results/part-00001/data/index due to exception: java.io.FileNotFoundException: File hdfs://...:8020/user/emre/outdir/results/part-00001/data/index does not exist.

Although file is exist in path hdfs://...:8020/user/emre/outdir/results/part-00001/data/index, i got this exception.

Emre,

Thanks

Re: Batch Indexing Using MapReduce to Solr

mdrob — Thu, 12 Nov 2015 19:59:02 GMT

Which version of Cloudera Search are you using?

That warning happens when trying to load some metrics on the data in HDFS. Are there any other warnings or errors in the logs that you can find?

This warning is probably safe to ignore in your case.

Re: Batch Indexing Using MapReduce to Solr

heerkek — Fri, 13 Nov 2015 11:51:03 GMT

I'm using search-1.0.0+cdh5.4.5+0 version.

I got another exception java.io.FileNotFoundException: File hdfs://...:8020/user/emre/outdir/results/part-00000/data/index does not exist.

But, related file is exist.

Thanks,

Emre

Re: Batch Indexing Using MapReduce to Solr

mdrob — Fri, 13 Nov 2015 14:51:04 GMT

Hi Emre,

Thanks for updating me on your version. That will help me find the cause of the problem.

Can you provide more context from your logs? This second exception looks like the same thing, but for a different task output.

Can you be more specific about what kind of failure you see?

Re: Batch Indexing Using MapReduce to Solr

heerkek — Fri, 13 Nov 2015 15:55:21 GMT

I cannot see another error in solr log.

MapReduceIndexerTool is working succesfully. Details =>

497 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Indexing 2 files using 2 real mappers into 2 reducers
20730 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Done. Indexing 2 files using 2 real mappers into 2 reducers took 6.7444393E9 secs
20741 [main] INFO org.apache.solr.hadoop.GoLive - Live merging of output shards into Solr cluster...
20744 [pool-4-thread-2] INFO org.apache.solr.hadoop.GoLive - Live merge hdfs://...:8020/user/emre/outdir/results/part-00001 into http://...:8983/solr
20744 [pool-4-thread-1] INFO org.apache.solr.hadoop.GoLive - Live merge hdfs://...:8020/user/emre/outdir/results/part-00000 into http://...:8983/solr
20885 [main] INFO org.apache.solr.hadoop.GoLive - Committing live merge...
20892 [main] INFO org.apache.solr.common.cloud.SolrZkClient - Using default ZkCredentialsProvider
20893 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Waiting for client to connect to ZooKeeper
20897 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager - Watcher org.apache.solr.common.cloud.ConnectionManager@61d818f6 name:ZooKeeperConnection Watcher:...:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
20897 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Client is connected to ZooKeeper
20897 [main] INFO org.apache.solr.common.cloud.SolrZkClient - Using default ZkACLProvider
20898 [main] INFO org.apache.solr.common.cloud.ZkStateReader - Updating cluster state from ZooKeeper...
20922 [main] INFO org.apache.solr.hadoop.GoLive - Done committing live merge
20922 [main] INFO org.apache.solr.hadoop.GoLive - Live merging of index shards into Solr cluster took 6.0152416E7 secs
20922 [main] INFO org.apache.solr.hadoop.GoLive - Live merging completed successfully
20922 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Succeeded with job: jobName: org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, jobId: job_1447398200936_0024
20922 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Success. Done. Program took 6.9784602E9 secs. Goodbye.

But, i cannot see any doc in solr index. When run solr query that t...:8983/solr/collection1/select?q=*%3A*&wt=json&indent=true , number of found is 0.

Re: Batch Indexing Using MapReduce to Solr

heerkek — Mon, 16 Nov 2015 13:06:27 GMT

I mistaken morphline configuration. Because of this mistake, docs could not indexed in solr.

Thanks for responces,

Emre