SmartSense bundle status is “Upload Failed” but the bundle is still visible in Datalake

[#1]

Affected versions:

1.4.0, 1.4.1, 1.4.2, 1.4.3

Symptoms:

A Bundle is available in Datalake but the status is still Upload Failed. Following error is reported in the hst-server.log.

2017-09-08 17:44:09,319 ERROR [pool-1-thread-1] UploadRunnable:236 -
Error occurred during upload bundlecom.hortonworks.support.tools.server.SystemException:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1
column 10   at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:182)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at com.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176)  at
com.hortonworks.support.tools.gateway.UploadRunnable.run(UploadRunnable.java:224)  at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at
java.lang.Thread.run(Thread.java:745)Caused by:
com.hortonworks.smartsense.gateway.client.GatewayClientException:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1
column 10  at
com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:246)  at
com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:125)  ... 26 moreCaused by: com.google.gson.JsonSyntaxException:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1
column 10  at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)  at
com.google.gson.Gson.fromJson(Gson.java:795)  at
com.google.gson.Gson.fromJson(Gson.java:761)  at
com.google.gson.Gson.fromJson(Gson.java:710)  at com.google.gson.Gson.fromJson(Gson.java:682)  at
com.hortonworks.smartsense.gateway.client.AGatewayClient.populateResponse(AGatewayClient.java:43)  at
com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:230)  ... 27 moreCaused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but
was STRING at line 1 column 10  at
com.google.gson.stream.JsonReader.expect(JsonReader.java:339)  at
com.google.gson.stream.JsonReader.beginObject(JsonReader.java:322)  at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165)  ... 33 more

Reason:

This happens due to incompatibility of HST Server and HST Gateway versions.

The upload response format sent by Hortonworks SmartSense Datalake Upload Service is updated in 1.4.x. HST Server expects are particular format of the response from the Upload Service. HST Server 1.4.x required HST Gateway 1.4.x to provide the right response.

Solution:

Verify the installed hst version on HST Server node and the Gateway Node.

Command : hst --version

If SMARTSENSE-HST rpm version on Gateway is lower than HST Server, then it has to be upgraded. HST Gateway is backward compatible with HST Server.