Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am writing code to process stream of data and write to hive tables, for this using hive streaming , Hcatalog but getting error Caused by: java.io.NotSerializableException: org.apache.hadoop.hive.conf.HiveConf

Highlighted

I am writing code to process stream of data and write to hive tables, for this using hive streaming , Hcatalog but getting error Caused by: java.io.NotSerializableException: org.apache.hadoop.hive.conf.HiveConf

New Contributor

I am writing code to process stream of data and write to hive tables, for this using hive streaming , Hcatalog but getting error Caused by: java.io.NotSerializableException: org.apache.hadoop.hive.conf.HiveConf

Sample:

import org.apache.hadoop.hive.conf.HiveConf

import org.apache.hive.hcatalog.streaming.StreamingConnection

val hcatconf: HiveConf = HiveHcat.prepareEnvironment()

val hcatconnection: StreamingConnection = null;

inputDirectStream.foreachRDD { rdd => rdd.foreachPartition { partitionOfCDCMessages => partitionOfCDCMessages.foreach{ //java goes here val mp:MessageProcessor = new MessageProcessor(sparksession, hcatconnection, hcatconf) message => mp.processMessage(message) } } }

(followig instruction given here: https://community.hortonworks.com/articles/49949/test-7.html )

but getting error like this:

Caused by: java.io.NotSerializableException: org.apache.hadoop.hive.conf.HiveConf Serialization stack: - object not serializable (class: org.apache.hadoop.hive.conf.HiveConf, value: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@662706a7, file:/etc/spark2/2.6.2.3-1/0/hive-site.xml) - field (class: com.mycompany.mypackage.Main$anonfun$main$1, name: hcatconf$1, type: class org.apache.hadoop.hive.conf.HiveConf) - object (class com.mycompany.mypackage.Main$anonfun$main$1, <function1>) - field (class: com.mycompany.mypackage.Main$anonfun$main$1$anonfun$apply$1, name: $outer, type: class com.xerox.CDCProcessor.Main$anonfun$main$1) - object (class com.mycompany.mypackage.Main$anonfun$main$1$anonfun$apply$1, <function1>)

please help

Don't have an account?
Coming from Hortonworks? Activate your account here