Hi @rki_ , Thanks for the explanation. I had hoped to have found a reason for the index writer closed error of SolR.   Thanks you anyway ... View more

Hi,   you can try disabling the network adapter from VM configs. ... View more

Hi,   aren't there networks problems? ... View more

The CDP platform is great if your use cases require it. I am noticing the issue, however, in CDP Public cloud implementation.   Have you tried it? ... View more

This is the error log from the "slave" node when knox is started. I have no idea why it is used knox to access to abfs, but it is consistent with the symptoms: 2021-03-08 16:15:28,657 ERROR idbroker.azure (KnoxMSICredentials.java:httpPatchRequest(416)) - Request to attach identities to VM failed with response code 400, message: {"error":{"code":"FailedIdentityOperation","message":"Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachines/xxx' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/msi/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mock-idbroker-admin-identity' was not found.\"}}'.'."}} 2021-03-08 16:15:28,658 ERROR idbroker.azure (KnoxAzureClient.java:addIdentitiesToVM(288)) - Error attaching identities to VM: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,658 ERROR idbroker.azure (KnoxAzureClient.java:generateAccessToken(425)) - Azure ADLS2, error obtaining access token, cause : java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,659 ERROR idbroker.azure (KnoxAzureClient.java:getCredentialsForRole(163)) - Azure ADLS2, error obtaining access token, cause : java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,661 ERROR idbroker.azure (KnoxAzureClient.java:getCredentialsForRole(164)) - StackTrace: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4873) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getCachedAccessToken(KnoxAzureClient.java:346) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getCredentialsForRole(KnoxAzureClient.java:127) at org.apache.knox.gateway.service.idbroker.AbstractKnoxCloudCredentialsClient.getCredentialsForRole(AbstractKnoxCloudCredentialsClient.java:119) at org.apache.knox.gateway.service.idbroker.KnoxCloudCredentialsClientManager.getCredentialsForRole(KnoxCloudCredentialsClientManager.java:43) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getRoleCredentialsResponse(IdentityBrokerResource.java:198) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:180) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:173) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:169) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentials(IdentityBrokerResource.java:137) at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:406) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:350) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:106) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:534) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:482) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:419) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.access$000(AbstractIdentityAssertionFilter.java:53) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter$1.run(AbstractIdentityAssertionFilter.java:161) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doAs(AbstractIdentityAssertionFilter.java:156) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:146) at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:94) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter$1.run(AbstractJWTFilter.java:207) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter.continueWithEstablishedSecurityContext(AbstractJWTFilter.java:202) at org.apache.knox.gateway.provider.federation.jwt.filter.JWTFederationFilter.doFilter(JWTFederationFilter.java:93) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:50) at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:167) at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:92) at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:135) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:41) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:106) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.generateAccessToken(KnoxAzureClient.java:426) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.lambda$getCachedAccessToken$0(KnoxAzureClient.java:350) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4878) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ... 107 more Caused by: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.addIdentitiesToVM(KnoxAzureClient.java:289) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.loadUserIdentities(KnoxAzureClient.java:186) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getAccessTokenUsingMSI(KnoxAzureClient.java:476) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.generateAccessToken(KnoxAzureClient.java:416) ... 113 more Caused by: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxMSICredentials.httpPatchRequest(KnoxMSICredentials.java:429) at org.apache.knox.gateway.service.idbroker.azure.KnoxMSICredentials.attachIdentities(KnoxMSICredentials.java:188) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.addIdentitiesToVM(KnoxAzureClient.java:256) ... 116 more ... View more

Hi @jessica moore, see this https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md ... View more