Hello,
OK, so things have moved on, and now I am hoping someone can help now things have moved on. I have created a mount point on the head node of my HDInsight cluster, this mount point goes to Azure Blob storage (File Share) and I am achieving partial backup, or at least this is what the error is telling me. I think what it really means is that it is not able to do any backups at all as I have yet to see anything appear in the blob storage.
I created a file share using steps from these URLs, the first tells me how to create a file share, the second tells me how to create mount point on an Linux (ubuntu) node to that file share.
https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share
https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-linux
I have tested the link to the file share by uploading a txt file to it, which then appears on the Linux node hosting the mount point.
If I execute the following line in the Solr UI, I receive the subsequent error...
http://1STPARTOFIPADDRESS.213:8983/solr/admin/collections?action=BACKUP&name=prod_cat5&collection=pr...
{
"responseHeader":{
"status":500,
"QTime":216},
"success":{
"1STPartofIPaddress.202:8983_solr":{
"responseHeader":{
"status":0,
"QTime":6}}},
"failure":{
"1STPartofIPaddress.207:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://1STPartofIPaddress.207:8983/solr: Failed to backup core=products_yan_1_replica_n4 because org.apache.solr.common.SolrException: Directory to contain snapshots doesn't exist: file:///mnt/solrbkp/prod_cat5"},
"Operation backup caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not backup all replicas",
"exception":{
"msg":"Could not backup all replicas",
"rspCode":500},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"Could not backup all replicas",
"trace":"org.apache.solr.common.SolrException: Could not backup all replicas\n\tat org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:258)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:230)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:530)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)\n\tat java.lang.Thread.run(Thread.java:748)\n",
"code":500}}
If I run the same backup command again, I receive...
{
"responseHeader":{
"status":400,
"QTime":145},
"Operation backup caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: The backup directory already exists: file:///mnt/solrbkp/prod_cat5/",
"exception":{
"msg":"The backup directory already exists: file:///mnt/solrbkp/prod_cat5/",
"rspCode":400},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"The backup directory already exists: file:///mnt/solrbkp/prod_cat5/",
"code":400}}So here it says the location already exists, but it does not exist in the blob store file share where I would expect to find it, and the previous error states it does not exist. So where is it? If I specify the backup name, is the last part of the desired backup location supposed to be created automatically or I'm I correct in specifying it?
Do I need to create the same mount point on multiple nodes? If so, just the nodes that are hosting collections/indexes/shards? Or also on the Head and Zookeeper nodes?
I have tested the link to the file share by uploading a txt file to it, which then appears on the Linux node hosting the mount point.
Thank you in advance for any help or advice.
Regards,
Paul.
