Hi Community, I'm running a basic spark job which reads from an HBase table. I can see the job is getting complete without any error, but in output I get the empty rows. Will appreciate any help. Below is my code object objectName { def catalog = s"""{ |"table":{"namespace":"namespaceName", "name":"tableName"}, |"rowkey":"rowKeyAttribute", |"columns":{ |"Key":{"cf":"rowkey", "col":"rowKeyAttribute", "type":"string"}, |"col1":{"cf":"cfName", "col":"col1", "type":"bigint"}, |"col2":{"cf":"cfName", "col":"col2", "type":"string"} |} |}""".stripMargin def main(args: Array[String]) { val spark = SparkSession.builder() .appName("dummyApplication") .getOrCreate() val sc = spark.sparkContext val sqlContext = spark.sqlContext import sqlContext.implicits._ def withCatalog(cat: String): DataFrame = { sqlContext .read .options(Map(HBaseTableCatalog.tableCatalog -> cat)) .format("org.apache.spark.sql.execution.datasources.hbase") .load() } } ... View more

Hi Community, I'm new to spark and have been struggling to execute a spark job which connects to a HBase table. In YARN GUI I can see the spark job is getting to state Running but then it fails with below error : 18/07/01 21:08:20 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6a8bcb640x0, quorum=localhost:2181, baseZNode=/hbase 18/07/01 21:08:20 INFO ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 18/07/01 21:08:20 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 18/07/01 21:08:20 WARN RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid ZooKeeper is up and running. Below is my code //Create table catalog object foo { def catalog = s"""{ |"table":{"namespace":"foo", "name":"bar"}, |"rowkey":"key", |"columns":{ |"col0":{"col":"key", "type":"string"}, |"col1":{"cf":"data", "col":"id", "type":"bigint"} |} |} "" ". stripMargin def main(args: Array[String]) { val spark = SparkSession.builder() .appName("foo") .getOrCreate() choose sc = kick.sparkContext val sqlContext = spark.sqlContext import sqlContext.implicits._ def withCatalog(cat: String): DataFrame = { sqlContext .read .options(Map(HBaseTableCatalog.tableCatalog -> cat)) .format("org.apache.spark.sql.execution.datasources.hbase") .load() } // Read from HBase table val df = withCatalog(catalog) df.show df.filter($"col0" === "1528801346000_200550232_2955") .select($"col0", $"col1").show spark.stop() } I'll really appreciate any help on this. I couldnt find any convincing answer in stackoverflow as well. ... View more

I have flattened a json using jolt since the input was json array it has resulted in list for the keys. But to use PutHBaseJson processor I need scalar values to define rowkey. Is there a way to repeat the whole json again and keep one value at a time? So that the uniqueness is maintained. Below are my input, transformation and output Input { "resource": { "id": "1234", "name": "Resourse_Name" }, "data": [ { "measurement": { "key": "5678", "value": "status" }, "timestamp": 1517784040000, "value": 1 }, { "measurement": { "key": "91011", "value": "location" }, "timestamp": 1519984070000, "value": 0 } ] } Transformation [ { "operation": "shift", "spec": { "resource": { "id": "resource_id", "name": "resource_name" }, // // Turn all the SecondaryRatings into prefixed data // like "rating-Design" : 4 "data": { "*": { // the "&" in "rating-&" means go up the tree 0 levels, // grab what is ther and subtitute it in "measurement": { "*": "measurement_&" }, "timestamp": "measurement_timestamp", "value": "value" } } } } ] Output { "resource_id" : "1234", "resource_name" : "Resourse_Name", "measurement_key" : [ "5678", "91011" ], "measurement_value" : [ "status", "location" ], "measurement_timestamp" : [ 1517784040000, 1519984070000 ], "value" : [ 1, 0 ] } Expected output { "resource_id" : "1234", "resource_name" : "Resourse_Name", "measurement_key" : "5678", "measurement_value" : "status", "measurement_timestamp" : 1517784040000, , "value" : 1 }, { "resource_id" : "1234", "resource_name" : "Resourse_Name", "measurement_key" : "91011" , "measurement_value" : "location" , "measurement_timestamp" : 1519984070000 , "value" : 0 } ... View more

I have a requirement where I need to create a hive table which uses the same data multiple times. In order to do that I'm having multiple replace text steps followed by a merge content. File content is as below : abc123 active true sometext I want to read the file as below into hive table : abc123;active;true;sometext;123;active;c1;true; In order to achieve this I'm using replace text where first I read as below(regex replace): abc123;active;true;sometext; Then 123;active; And finally as c1;true; But I'm not able to merge content horizontally, is there a way to do it ? Or may be a easier way to achieve the same result. Any help will be appriciated. Thanks. ... View more