Hi,   I have never encountered problems on Centos7 and jdk8 for Zeppelin 0.7.3. But the requirement in the doc could also depend on some specific interpreters. We mostly used Hive interpreter.   I would try with a dedicated virtula nodes for testing. ... View more

Hi,   I don't think there is a way to retrieve this information via API rest.     Make a python script, from there you should be able to retrieve the quota (https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html).   You can try to configure a custom ambari alert whit the script.   Let me know if you can because it could be useful to many people. ... View more

Hi,   can you try to force hive compaction? ALTER TABLE <db_name>.<table_name> PARTITION (<partition_name>='<partition_value>') COMPACT 'major'; with the show compactions command it checks when it has finished and the result (failed or ok).   After this it tries to execute the count again   ... View more

Hi, The tables where you count in your example, are both on the new cluster, correct?   Are the tables ACID? What version of hive are you using?   ... View more

The CDP platform is great if your use cases require it. I am noticing the issue, however, in CDP Public cloud implementation.   Have you tried it? ... View more

This is the error log from the "slave" node when knox is started. I have no idea why it is used knox to access to abfs, but it is consistent with the symptoms: 2021-03-08 16:15:28,657 ERROR idbroker.azure (KnoxMSICredentials.java:httpPatchRequest(416)) - Request to attach identities to VM failed with response code 400, message: {"error":{"code":"FailedIdentityOperation","message":"Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachines/xxx' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/msi/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mock-idbroker-admin-identity' was not found.\"}}'.'."}} 2021-03-08 16:15:28,658 ERROR idbroker.azure (KnoxAzureClient.java:addIdentitiesToVM(288)) - Error attaching identities to VM: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,658 ERROR idbroker.azure (KnoxAzureClient.java:generateAccessToken(425)) - Azure ADLS2, error obtaining access token, cause : java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,659 ERROR idbroker.azure (KnoxAzureClient.java:getCredentialsForRole(163)) - Azure ADLS2, error obtaining access token, cause : java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden 2021-03-08 16:15:28,661 ERROR idbroker.azure (KnoxAzureClient.java:getCredentialsForRole(164)) - StackTrace: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4873) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getCachedAccessToken(KnoxAzureClient.java:346) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getCredentialsForRole(KnoxAzureClient.java:127) at org.apache.knox.gateway.service.idbroker.AbstractKnoxCloudCredentialsClient.getCredentialsForRole(AbstractKnoxCloudCredentialsClient.java:119) at org.apache.knox.gateway.service.idbroker.KnoxCloudCredentialsClientManager.getCredentialsForRole(KnoxCloudCredentialsClientManager.java:43) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getRoleCredentialsResponse(IdentityBrokerResource.java:198) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:180) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:173) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentialsResponse(IdentityBrokerResource.java:169) at org.apache.knox.gateway.service.idbroker.IdentityBrokerResource.getCredentials(IdentityBrokerResource.java:137) at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:406) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:350) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:106) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:534) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:482) at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:419) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.access$000(AbstractIdentityAssertionFilter.java:53) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter$1.run(AbstractIdentityAssertionFilter.java:161) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doAs(AbstractIdentityAssertionFilter.java:156) at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:146) at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:94) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter$1.run(AbstractJWTFilter.java:207) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter.continueWithEstablishedSecurityContext(AbstractJWTFilter.java:202) at org.apache.knox.gateway.provider.federation.jwt.filter.JWTFederationFilter.doFilter(JWTFederationFilter.java:93) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:50) at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:349) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:263) at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:167) at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:92) at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:135) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:41) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:106) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.generateAccessToken(KnoxAzureClient.java:426) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.lambda$getCachedAccessToken$0(KnoxAzureClient.java:350) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4878) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ... 107 more Caused by: java.lang.RuntimeException: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.addIdentitiesToVM(KnoxAzureClient.java:289) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.loadUserIdentities(KnoxAzureClient.java:186) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.getAccessTokenUsingMSI(KnoxAzureClient.java:476) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.generateAccessToken(KnoxAzureClient.java:416) ... 113 more Caused by: javax.ws.rs.WebApplicationException: HTTP 403 Forbidden at org.apache.knox.gateway.service.idbroker.azure.KnoxMSICredentials.httpPatchRequest(KnoxMSICredentials.java:429) at org.apache.knox.gateway.service.idbroker.azure.KnoxMSICredentials.attachIdentities(KnoxMSICredentials.java:188) at org.apache.knox.gateway.service.idbroker.azure.KnoxAzureClient.addIdentitiesToVM(KnoxAzureClient.java:256) ... 116 more ... View more

Hi @jessica moore, see this https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md ... View more