Affected versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3 Symptoms: A Bundle is available in Datalake but the status is still Upload Failed. Following error is reported in the hst-server.log. 2017-09-08 17:44:09,319 ERROR [pool-1-thread-1] UploadRunnable:236 - Error occurred during upload bundlecom.hortonworks.support.tools.server.SystemException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:182) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) at com.hortonworks.support.tools.gateway.UploadRunnable.run(UploadRunnable.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: com.hortonworks.smartsense.gateway.client.GatewayClientException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:246) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:125) ... 26 moreCaused by: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:795) at com.google.gson.Gson.fromJson(Gson.java:761) at com.google.gson.Gson.fromJson(Gson.java:710) at com.google.gson.Gson.fromJson(Gson.java:682) at com.hortonworks.smartsense.gateway.client.AGatewayClient.populateResponse(AGatewayClient.java:43) at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:230) ... 27 moreCaused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 10 at com.google.gson.stream.JsonReader.expect(JsonReader.java:339) at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:322) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 33 more Reason: This happens due to incompatibility of HST Server and HST Gateway versions. The upload response format sent by Hortonworks SmartSense Datalake Upload Service is updated in 1.4.x. HST Server expects are particular format of the response from the Upload Service. HST Server 1.4.x required HST Gateway 1.4.x to provide the right response. Solution: Verify the installed hst version on HST Server node and the Gateway Node. Command : hst --version If SMARTSENSE-HST rpm version on Gateway is lower than HST Server, then it has to be upgraded. HST Gateway is backward compatible with HST Server. ... View more

Affected versions: 1.2.x Symptoms: Ambari was running as root during installation, then after configured for non-root, the issue occurs. When we create a bundle, we are not able to encrypt it because ambari is not able to read the public key under /var/lib/smartsense/hst-common/encryption/keys. File permissions [root@DPDev hdfs]# ls -lsR /var/lib/smartsense/hst-common /var/lib/smartsense/hst-common:total 8 4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 anonymization 4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 encryption /var/lib/smartsense/hst-common/anonymization:total 4 4 drwxr-xr-x 2 ambari ambari 4096 Apr 20 18:21 keys /var/lib/smartsense/hst-common/anonymization/keys:total 8 4 -rw-r--r-- 1 root root 44 Apr 20 16:59 private_anonymization.key 4 -rw-r--r-- 1 root root 44 Apr 20 18:01 shared_anonymization.key /var/lib/smartsense/hst-common/encryption:total 4 4 drw-r----- 2 ambari ambari 4096 Apr 20 16:59 keys /var/lib/smartsense/hst-common/encryption/keys:total 8 4 -rw-r----- 1 ambari ambari 216 Mar 10 12:52 public.key4 -rw-r----- 1 ambari ambari 216 Mar 10 12:52 test_public.key Encrypted download error The webpage at http://10.20.30.40:8080/api/v1/views/HORTONWORKS_SMARTSENSE/versions/1.2.1.0-70/instances/SmartSense/resources/hst/bundles/a-00000000-c-00000000_dpdev_0_2016-04-20_18-06-04?format=download&encrypt=true might be temporarily down or it may have moved permanently to a new web address.ERR_INVALID_RESPONSE hst-server.out or hst-server.log Apr 20, 2016 6:46:45 PM com.hortonworks.support.tools.server.streaming.EncryptedStreamingOutput write INFO: Encrypting file : /var/lib/smartsense/hst-server/data/a-00000000-c-00000000_dpdev_0_2016-04-20_18-21-02.tgz Apr 20, 2016 6:46:45 PM com.hortonworks.smartsense.anonymization.Configuration loadKey INFO: Path /var/lib/smartsense/hst-common/encryption/keys/public.key doesn't exists or not a file to load key. Apr 20, 2016 6:46:45 PM com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto encrypt SEVERE: Invalid key: No installed provider supports this key: (null). Apr 20, 2016 6:46:45 PM org.eclipse.jetty.servlet.ServletHandler doHandle WARNING: /api/v1/bundles/a-00000000-c-00000000_dpdev_0_2016-04-20_18-21-02 com.hortonworks.smartsense.anonymization.crypto.CryptoException: No installed provider supports this key: (null) at com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto.encrypt(RSACrypto.java:106) at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.encrypt(ArchiveFileCrypto.java:159) at com.hortonworks.smartsense.anonymization.crypto.FileEncryptor.encrypt(FileEncryptor.java:75) at com.hortonworks.support.tools.server.streaming.EncryptedStreamingOutput.write(EncryptedStreamingOutput.java:61) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at com.hortonworks.support.tools.server.security.authorization.SupportToolAuthorizationFilter.doFilter(SupportToolAuthorizationFilter.java:93) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) at com.hortonworks.support.tools.servlet.SupportToolPersistFilter.doFilter(SupportToolPersistFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:363) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.chooseProvider(Cipher.java:893) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at com.hortonworks.smartsense.anonymization.crypto.impl.RSACrypto.encrypt(RSACrypto.java:102) ... 64 more Reason: The permission for the public key under /var/lib/smartsense/hst-common/encryption/keys folder is 640. So only root could read it. As ambari was running as non-root user, it could not access this folder. Solution: Executed the command below and download the encrypted bundle again or re-upload the bundle. chmod -R 750 /var/lib/smartsense/hst-common/encryption ... View more

Affected versions: 1.4.x Symptoms: New recommendations are not displayed in the recommendations tab in the SmartSense view. There is a failure in hst-server.log : 2017-04-20 17:20:11,904 INFO [RecommendationAutoDownloadTask] GatewayClientHelper:74 - HTTP/1.1 200 OK 2017-04-20 17:20:11,904 INFO [RecommendationAutoDownloadTask] GatewayClientHelper:75 - OK 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] ArchiveFileCrypto:240 - Invalid key. Illegal key size or default parameters 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] RecommendationAutoDownloadTask:41 - Failed to download recommendations. Illegal key size or default parameters com.hortonworks.smartsense.anonymization.crypto.CryptoException: Illegal key size or default parameters at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:241) at com.hortonworks.smartsense.anonymization.crypto.FileDecryptor.decrypt(FileDecryptor.java:69) at com.hortonworks.support.tools.server.keygen.EncryptionKeyPairManager.decrypt(EncryptionKeyPairManager.java:109) at com.hortonworks.support.tools.gateway.GatewayClientHelper.getRecommendations(GatewayClientHelper.java:79) at com.hortonworks.support.tools.server.recommendation.RecommendationGatewayImpl.gatewayDownloadRecommendations(RecommendationGatewayImpl.java:66) at com.hortonworks.support.tools.server.BaseControllerModule$1.gatewayDownloadRecommendations(BaseControllerModule.java:83) at com.hortonworks.support.tools.server.recommendation.RecommendationGatewayImpl.download(RecommendationGatewayImpl.java:50) at com.hortonworks.support.tools.server.recommendation.RecommendationAutoDownloadTask.downloadRecommendations(RecommendationAutoDownloadTask.java:55) at com.hortonworks.support.tools.server.recommendation.RecommendationAutoDownloadTask.run(RecommendationAutoDownloadTask.java:39) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.security.InvalidKeyException: Illegal key size or default parameters at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1026) at javax.crypto.Cipher.implInit(Cipher.java:801) at javax.crypto.Cipher.chooseProvider(Cipher.java:864) at javax.crypto.Cipher.init(Cipher.java:1249) at javax.crypto.Cipher.init(Cipher.java:1186) at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:233) Reason: Hortonworks Datalake uses longer keys to encrypt the recommendations for more secure communication. JCE Unlimited strength is required to decrypt the received recommendations. This above provided exception will typically occur if JCE unlimited strength is not available or used in the HST Server node. Solution: Install JCE Unlimited on HST Server for the available JDK version and restart HST Server. If JCE Unlimited is already available, then verify following: JCE requires these 2 jars in JDK_HOME/jre/lib/security/ 1. US_export_policy.jar 2. local_policy.jar Right way to verify is to download the jars from Oracle site for a given jdk version and verify their checksum. Sometimes, these jars may be available in this dir but they are probably for a different jdk version. ... View more

Affected versions: 1.2.0, 1.2.1 Symptoms: Bundle upload fails with following error in hst-server.log: 10 Aug 2016 16:56:17,131 INFO [pool-1-thread-1] GatewayHttpsClient:219 - Executing Request:POST https://gateway0host.gateway.com:9451/api/v1/upload/bundle HTTP/1.110 Aug 2016 16:56:17,142 ERROR [pool-1-thread-1] GatewayHttpsClient:241 - SSL error occurred. Cleaning up certificates/keys at client side is required.10 Aug 2016 16:56:17,142 WARN [pool-1-thread-1] UploadRunnable:106 - SSL error occurred while connecting to gateway. Reseting local keys and certificates. Reason: Support for using md5 algorithm to setup 2-way SSL was stopped in JDK 8. We had md5 as our default algorithm in SmartSense 1.2.1 and lower. This causes conflict in SSL setup hst-server jdk version is lower than JDK 8. Solution: This happens when HST is java version 1.6 and Gateway is on version 1.8. This is fixed in 1.2.2. Upgrade HST to version higher than 1.2.1. Upgrading Gateway mostly will not cause any impact because it provides backward compatibility with respoect to HST Server. If, SmartSense upgrade is not possible then try to sync up the JDK versions between Gateway host and HST Server host. Similar issue can occur in 1.2.1 if the hst-agents and hst-server have similar mismatch in installed JDK versions. More details and steps to fix this manually are provided here : https://community.hortonworks.com/articles/24928/smartsense-agent-dying.html ... View more

Affected versions: 1.2.x, 1.3.0 Symptoms: Standalone Gateway is up and running. But still there is a failure pop up when trying to upload a bundle. Error details are provided in logs below: ##hst-server.log Caused by: com.hortonworks.smartsense.gateway.client.GatewayClientException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 7 at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:249) at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:100)... 12 more Caused by: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 7 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:795) at com.google.gson.Gson.fromJson(Gson.java:761) at com.google.gson.Gson.fromJson(Gson.java:710) at com.google.gson.Gson.fromJson(Gson.java:682) at com.hortonworks.smartsense.gateway.client.security.ClientCertificateManager.reqSignCert(ClientCertificateManager.java:159) at com.hortonworks.smartsense.gateway.client.security.ClientCertificateManager.checkSecuritySetup(ClientCertificateManager.java:190) at com.hortonworks.smartsense.gateway.client.https.HttpsClientFactory.verifyTwoWaySSLSetup(HttpsClientFactory.java:80) at com.hortonworks.smartsense.gateway.client.https.HttpsClientFactory.getSSLHttpClient(HttpsClientFactory.java:114) at com.hortonworks.smartsense.gateway.client.https.HttpsClientFactory.getHttpsClient(HttpsClientFactory.java:69) at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:190)... 13 more Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 7 at com.google.gson.stream.JsonReader.expect(JsonReader.java:339) ##hst-gateway.log 17 Oct 2016 14:13:39,820 INFO qtp1898155970-67 GatewaySecurityFilter:71 - Filtering https://770745-hadoopmaster03.abcxyz.com:9450/certs/770745-hadoopmaster03.abcxyz.com for security purposes17 Oct 2016 14:13:39,821 WARN qtp1898155970-67 GatewaySecurityFilter:125 - Request https://770745-hadoopmaster03.abcxyz.com:9450/certs/770745-hadoopmaster03.abcxyz.com doesn't match any pattern.17 Oct 2016 14:13:39,821 WARN qtp1898155970-67 GatewaySecurityFilter:77 - This request is not allowed on this port: https://770745-hadoopmaster03.abcxyz.com:9450/certs/770745-hadoopmaster03.abcxyz.com ##Popup while clicking on "upload". Unable to connect to the on-premise SmartSense Gateway (host: 770745-hadoopmaster03.abcxyz.com, port: 9451). Please verify the gateway.host and gateway.port properties in the HST Server configurations are set correctly and the on-premise SmartSense Gateway is up and running. Reason: Gateway had a filter which would reject any requests if the host url started with a number. Solution: The pattern matcher for host in the url is fixed in 1.3.1. Upgrade to 1.3.1 or rename the gateway host to remove the preceding numbers and restart the gateway. SmartSense 1.3.1 upgrade documents : http://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.3.1/bk_installation/content/upgrade_scenarios.html ... View more

Affected versions: SmartSense 1.3.x Symptoms: Gateway start fails with following exception in hst-gateway.log : INFO 2016-10-21 12:32:42,881 hst-gateway.py:520 - SmartSense Gateway is not running... INFO 2016-10-21 12:32:42,882 hst-gateway.py:539 - Running server: ['/bin/sh', '-c', 'ulimit -n 10000; /usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dlog.file.name=hst-gateway.log -Xms512m -Xmx1024m -cp /etc/hst/conf:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.gateway.server.GatewayServer >/var/log/hst/hst-gateway.out 2>&1 &'] INFO 2016-10-21 12:32:42,884 hst-gateway.py:542 - Executing Gateway Server start command 21 Oct 2016 12:32:43,505 DEBUG main GatewayConfigValidator:40 - Found supported transfer type:HTTPS 21 Oct 2016 12:32:43,507 INFO main GatewayConfigValidator:67 - Found configurations to be used for connectivity. 21 Oct 2016 12:32:43,509 INFO main GatewayValidator:149 - Trying to reach: smartsense-dev.hortonworks.com:443 21 Oct 2016 12:32:43,667 ERROR main GatewayValidator:93 - Failed to reach SmartSense landing zone host:smartsense-dev.hortonworks.com, port:443 21 Oct 2016 12:32:43,667 INFO main GatewayValidator:149 - Trying to reach: hortonworks.com:80 21 Oct 2016 12:32:43,678 ERROR main GatewayValidator:100 - Failed to reach hortonworks.com:80. 21 Oct 2016 12:32:43,679 ERROR main GatewayServer:78 - Error occured during Gateway Server Setup. Unable to start Gateway Server. com.hortonworks.smartsense.gateway.GatewayException: Unable to reach hortonworks.com:80. Please verify outbound connectivity on Gateway. at com.hortonworks.smartsense.gateway.server.GatewayValidator.validateReachable(GatewayValidator.java:102) at com.hortonworks.smartsense.gateway.server.GatewayValidator.validateSetup(GatewayValidator.java:65)at com.hortonworks.smartsense.gateway.server.GatewayServer.run(GatewayServer.java:99) at com.hortonworks.smartsense.gateway.server.GatewayServer.main(GatewayServer.java:76) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.hortonworks.smartsense.gateway.server.GatewayValidator.isReachable(GatewayValidator.java:150) at com.hortonworks.smartsense.gateway.server.GatewayValidator.validateReachable(GatewayValidator.java:90)... 3 more Reason: There is a gateway validation process that checks for outbound connectivity during start up. This validation test was trying an outbound socket connectivity check without passing through the proxy and failing. Solution: Fixed in SmartSense 1.4.0. Please upgrade to this version and restart Gateway to automatically fix the issue. Gateway supports backward compatibility where HST 1.2.x or HST 1.3.x can use the Gateway 1.4.0 to upload bundles. Otherwise, for 1.3.x set the property in /etc/hst/conf/hst-gateway.ini in gateway section and restart Gateway: start.validation.enabled=false ... View more

Affected versions: SmartSense 1.2.x, 1.3.x with Ambari lower than 2.4.1 Symptoms: Users / Groups are pre-created before setting up the cluster. As such, ignore_groupsusers_create needs to be set to true so groups are not modified when services are added. This works fine for all components except smartsense which fails with error performing usermod : resource_management.core.exceptions.Fail: Execution of 'usermod -G -adp-hdfs -g adp-hadoop adp-xxxx' returned 6. usermod: group 'adp-hdfs' does not exist This group does NOT exist. Note, within the stdout you do actually see the following message: Skipping creation of User and Group as host is sys prepped or ignore_groupusers_create flag is on Reason: The “ignore_groupsusers_create” configurations was not considered while adding / installing SmartSense service. Solution: New fix is available in Ambari 2.4.2.6-5 or higher. Upgrade is the cleanest solution in this case. ... View more

Affected versions: All SmartSense versions Symptoms: YARN and MR dashboards work fine but there is no data for HDFS dashboard. Activity Analyzer logs are available at /var/log/smartsense-activity/activity-analyzer.log. These logs will shows an error stack trace similar to something as provided below: ERROR [pool-14-thread-1] ActivityManager:89 - Failed to process activity id /hadoop/hdfs/namenode/current/fsimage_0009281 of type HDFScom.hortonworks.smartsense.activity.ActivityException: Failed to process activity. Activity Type: HDFS; ActivityId: /hadoop/hdfs/namenode/current/fsimage_0009281; ActivityData: null at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:104) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:80) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Caused by: com.hortonworks.smartsense.activity.ActivityException: Error while getting fsImage iterator from path /hadoop/hdfs/namenode/current/fsimage_0009281 at org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageParser.getIterator(PBImageParser.java:118) at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:88) ... 6 more Caused by: java.io.IOException: Unsupported layout version -63 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) Reason: Activity Analyzer maintains all the dashboard in a new schema created in the same database as setup by Ambari Metrics Collector. It also utilizes the clients for HDFS available in the AMS libraries. This issue can happen if Activity Analyzer is used with older Hadoop/MapReduce/YARN libraries from AMS. Solution: Verify all the jars available under /usr/lib/ambari-metrics-collector and remove the older versions. Restart the Activity Analyzer after removing the older jars. Examples of older version jars: /usr/lib/ambari-metrics-collector/hadoop-annotations-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-auth-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-client-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-common-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.6.0.2.2.1.0-2340.jar /usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar /usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.6.0.2.2.0.0-2041.jar ... View more