Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

Solved Go to solution

Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

New Contributor
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

I don't think you can store the index on local and HDFS at the same time. The location of the index is based on which directory factory is being used, and in the case of HDFS you would be using the HDFS directory factory which would only be storing the index on HDFS. See the following:

https://cwiki.apache.org/confluence/display/solr/DataDir+and+DirectoryFactory+in+SolrConfig

https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS

2 REPLIES 2

Re: Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

I don't think you can store the index on local and HDFS at the same time. The location of the index is based on which directory factory is being used, and in the case of HDFS you would be using the HDFS directory factory which would only be storing the index on HDFS. See the following:

https://cwiki.apache.org/confluence/display/solr/DataDir+and+DirectoryFactory+in+SolrConfig

https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS

Highlighted

Re: Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

New Contributor

Can we store index in local file system as well as in HDFS? Simultaneaously.

Solr supports both indices on local file system and HDFS. It just depends on the directory factory being used for the collection. A single collection cannot span different file systems. What should be possible (haven't tested) is to have say two collections (one local filesystem, one hdfs filesystem) and then use Solr collection aliases to search both collections at once.

Also can we set a custom job periodically to upload index from local file system to HDFS?

There is nothing different between an index on HDFS and a local index. Moving an index between the two can be done carefully making sure that the correct index ends up in the right location. If the index is moved improperly (ie: shards don't line up) then you will get bad results.

Don't have an account?
Coming from Hortonworks? Activate your account here