I have batch data stored in Hive and realtime streaming data stored in HBase. I would like to create a view in Hive which joins a table in Hive with data in HBase. Using Hive on HBase is extremely slow. Is there a better way to accomplish this?
... View more
Sorry, the actual format is all comma separated. So after a fixed number of keys (let's assume eight) the pattern then switches to a dynamic number of key\value pairs: k,k,k,k,k,k,k,k,k,v,k,v,k,v,k,v.....
... View more
I have a file with the following pattern: K,K,K,K,K,K,KV,KV,KV,KV..... The initial standalone Keys (K) values are static and never change. The KV (key value pairs) after the keys are dynamic (additional KV pairs can be added or removed at anytime) and need to be listed as a map in Hive. The first K values would be listed as columns. Does someone have any code for a custom SerDe I can include in the Hive table definition for a file with this structure? Currently we are using a custom UDF with Python but would like to store the files directly in HDFS and only apply the schema at runtime.
... View more
The curl command is written for an "installed" state. Do you know what ServiceInfo state is for a pending install? I tried "PENDING" and that's not a valid value.
... View more
During the install of SmartSense we are getting the following errors: 500 status code received on GET method for API: /api/v1/clusters/test/services?ServiceInfo/state=INSTALLED&ServiceInfo/service_name.in(SMARTSENSE)¶ms/run_smoke_test=true¶ms/reconfigure_client=false Error message: org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Invalid transition for service, clusterName=test, clusterId=2, serviceName=SMARTSENSE, currentDesiredState=INIT, newDesiredState=STARTED { "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Invalid 'kdc_type' value: Existing MIT KDC" }
... View more
Running a Spark command to read a file and get an illegalArgumentException. This is HDP 2.3.1 and Spark 1.4.1. Same error occurs with PySpark. The error appears to come from the SnappyCompressionCodec. scala> var file = sc.textFile("hdfs://HdpTest:8020/user/weli/README.md") java.lang.IllegalArgumentException at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
... View more
