Can we store index in local file system as well as in HDFS? Simultaneaously. Also can we set a custom job periodically to upload index from local file system to HDFS?

[#1]

@Neeraj Sabharwal

[#2 · Accepted solution]

I don't think you can store the index on local and HDFS at the same time. The location of the index is based on which directory factory is being used, and in the case of HDFS you would be using the HDFS directory factory which would only be storing the index on HDFS. See the following:

https://cwiki.apache.org/confluence/display/solr/DataDir+and+DirectoryFactory+in+SolrConfig

https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS

[#3]

Can we store index in local file system as well as in HDFS? Simultaneaously.

Solr supports both indices on local file system and HDFS. It just depends on the directory factory being used for the collection. A single collection cannot span different file systems. What should be possible (haven't tested) is to have say two collections (one local filesystem, one hdfs filesystem) and then use Solr collection aliases to search both collections at once.

Also can we set a custom job periodically to upload index from local file system to HDFS?

There is nothing different between an index on HDFS and a local index. Moving an index between the two can be done carefully making sure that the correct index ends up in the right location. If the index is moved improperly (ie: shards don't line up) then you will get bad results.