Affected versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3 Symptoms: A Bundle is available in Datalake but the status is still Upload Failed. Following error is reported in the hst-server.log. 2017-09-08 17:44:09,319 ERROR [pool-1-thread-1] UploadRunnable:236 - Error occurred during upload bundlecom.hortonworks.support.tools.server.SystemException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:182) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.run(UploadRunnable.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: com.hortonworks.smartsense.gateway.client.GatewayClientException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:246) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:125) ... 26 moreCaused by: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:795) at com.google.gson.Gson.fromJson(Gson.java:761) at com.google.gson.Gson.fromJson(Gson.java:710) at com.google.gson.Gson.fromJson(Gson.java:682) at com.hortonworks.smartsense.gateway.client.AGatewayClient.populateResponse(AGatewayClient.java:43) at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:230) ... 27 moreCaused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.google.gson.stream.JsonReader.expect(JsonReader.java:339) at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:322) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 33 more Reason: This happens due to incompatibility of HST Server and HST Gateway versions. The upload response format sent by Hortonworks SmartSense Datalake Upload Service is updated in 1.4.x. HST Server expects are particular format of the response from the Upload Service. HST Server 1.4.x required HST Gateway 1.4.x to provide the right response. Solution: Verify the installed hst version on HST Server node and the Gateway Node. Command : hst --version If SMARTSENSE-HST rpm version on Gateway is lower than HST Server, then it has to be upgraded. HST Gateway is backward compatible with HST Server. ... View more

Affected versions: 1.2.x Symptoms: Ambari was running as root during installation, then after configured for non-root, the issue occurs. When we create a bundle, we are not able to encrypt it because ambari is not able to read the public key under /var/lib/smartsense/hst-common/encryption/keys. File permissions [root@DPDev hdfs]# ls -lsR /var/lib/smartsense/hst-common /var/lib/smartsense/hst-common:total 8 4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 anonymization 4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 encryption /var/lib/smartsense/hst-common/anonymization:total 4 4 drwxr-xr-x 2 ambari ambari 4096 Apr 20 18:21 keys /var/lib/smartsense/hst-common/anonymization/keys:total 8 4 -rw-r--r-- 1 root root 44 Apr 20 16:59 private_anonymization.key 4 -rw-r--r-- 1 root root 44 Apr 20 18:01 shared_anonymization.key /var/lib/smartsense/hst-common/encryption:total 4 4 drw-r----- 2 ambari ambari 4096 Apr 20 16:59 keys /var/lib/smartsense/hst-common/encryption/keys:total 8 4 -rw-r----- 1 ambari ambari 216 Mar 10 12:52 public.key4 -rw-r----- 1 ambari ambari 216 Mar 10 12:52 test_public.key Encrypted download error The webpage at http://10.20.30.40:8080/api/v1/views/HORTONWORKS_SMARTSENSE/versions/1.2.1.0-70/instances/SmartSense/resources/hst/bundles/a-00000000-c-00000000_dpdev_0_2016-04-20_18-06-04?format=download&encrypt=true might be temporarily down or it may have moved permanently to a new web address.ERR_INVALID_RESPONSE hst-server.out or hst-server.log Apr 20, 2016 6:46:45 PM com.hortonworks.support.tools.server.streaming.EncryptedStreamingOutput write INFO: Encrypting file : /var/lib/smartsense/hst-server/data/a-00000000-c-00000000_dpdev_0_2016-04-20_18-21-02.tgz Apr 20, 2016 6:46:45 PM com.hortonworks.smartsense.anonymization.Configuration loadKey INFO: Path /var/lib/smartsense/hst-common/encryption/keys/public.key doesn't exists or not a file to load key. Apr 20, 2016 6:46:45 PM com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto encrypt SEVERE: Invalid key: No installed provider supports this key: (null). Apr 20, 2016 6:46:45 PM org.eclipse.jetty.servlet.ServletHandler doHandle WARNING: /api/v1/bundles/a-00000000-c-00000000_dpdev_0_2016-04-20_18-21-02 com.hortonworks.smartsense.anonymization.crypto.CryptoException: No installed provider supports this key: (null) at com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto.encrypt(RSACrypto.java:106) at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.encrypt(ArchiveFileCrypto.java:159) at com.hortonworks.smartsense.anonymization.crypto.FileEncryptor.encrypt(FileEncryptor.java:75) at com.hortonworks.support.tools.server.streaming.EncryptedStreamingOutput.write(EncryptedStreamingOutput.java:61) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at com.hortonworks.support.tools.server.security.authorization.SupportToolAuthorizationFilter.doFilter(SupportToolAuthorizationFilter.java:93) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) at com.hortonworks.support.tools.servlet.SupportToolPersistFilter.doFilter(SupportToolPersistFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:363) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.chooseProvider(Cipher.java:893) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto.encrypt(RSACrypto.java:102) ... 64 more Reason: The permission for the public key under /var/lib/smartsense/hst-common/encryption/keys folder is 640. So only root could read it. As ambari was running as non-root user, it could not access this folder. Solution: Executed the command below and download the encrypted bundle again or re-upload the bundle. chmod -R 750 /var/lib/smartsense/hst-common/encryption ... View more

Affected versions: 1.4.x Symptoms: New recommendations are not displayed in the recommendations tab in the SmartSense view. There is a failure in hst-server.log : 2017-04-20 17:20:11,904 INFO [RecommendationAutoDownloadTask] GatewayClientHelper:74 - HTTP/1.1 200 OK 2017-04-20 17:20:11,904 INFO [RecommendationAutoDownloadTask] GatewayClientHelper:75 - OK 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] ArchiveFileCrypto:240 - Invalid key. Illegal key size or default parameters 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] RecommendationAutoDownloadTask:41 - Failed to download recommendations. Illegal key size or default parameters com.hortonworks.smartsense.anonymization.crypto.CryptoException: Illegal key size or default parameters at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:241) at com.hortonworks.smartsense.anonymization.crypto.FileDecryptor.decrypt(FileDecryptor.java:69) at com.hortonworks.support.tools.server.keygen.EncryptionKeyPairManager.decrypt(EncryptionKeyPairManager.java:109) at com.hortonworks.support.tools.gateway.GatewayClientHelper.getRecommendations(GatewayClientHelper.java:79) at com.hortonworks.support.tools.server.recommendation.RecommendationGatewayImpl.gatewayDownloadRecommendations(RecommendationGatewayImpl.java:66) at com.hortonworks.support.tools.server.BaseControllerModule$1.gatewayDownloadRecommendations(BaseControllerModule.java:83) at com.hortonworks.support.tools.server.recommendation.RecommendationGatewayImpl.download(RecommendationGatewayImpl.java:50) at com.hortonworks.support.tools.server.recommendation.RecommendationAutoDownloadTask.downloadRecommendations(RecommendationAutoDownloadTask.java:55) at com.hortonworks.support.tools.server.recommendation.RecommendationAutoDownloadTask.run(RecommendationAutoDownloadTask.java:39) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.security.InvalidKeyException: Illegal key size or default parameters at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1026) at javax.crypto.Cipher.implInit(Cipher.java:801) at javax.crypto.Cipher.chooseProvider(Cipher.java:864) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:233) Reason: Hortonworks Datalake uses longer keys to encrypt the recommendations for more secure communication. JCE Unlimited strength is required to decrypt the received recommendations. This above provided exception will typically occur if JCE unlimited strength is not available or used in the HST Server node. Solution: Install JCE Unlimited on HST Server for the available JDK version and restart HST Server. If JCE Unlimited is already available, then verify following: JCE requires these 2 jars in JDK_HOME/jre/lib/security/ 1. US_export_policy.jar 2. local_policy.jar Right way to verify is to download the jars from Oracle site for a given jdk version and verify their checksum. Sometimes, these jars may be available in this dir but they are probably for a different jdk version. ... View more