There are two versions of Spark in HDP 2.5, Spark 1.6 and Spark 2.0. I don't know how I can specify the version of Spark to be used. Can anyone advise me how to do that? Ambari admin console? Also I would like to submit job to Spark 2.0 from my application instead of spark-submit. What should I specify for the master url in the new SparkSession? Thanks. Donald ... View more

I have written one Avro ser/des library for Hive to read/write Avro binary data file, which does not contain the Avro schema. And I copied this library into $HIVE/lib. When I query the Hive external table, which uses my customized ser/des library, (e.g., select * from user_avro), it works as expected. However if I insert data into that Hive external table like "INSERT OVERWRITE TABLE user_avro SELECT * FROM user), it complains that the my customized ser/des class cannot be found. This output difference from Hive console is that the second one is launching MapReduce job. Does anyone know where I need to copy this customized ser/des library except $HIVE/lib? Many thanks in advance. ... View more

I have HDP 2.3.2 Sandbox. Currently I am learning Kafka and following the Hortonwok Kafka tutorial, Real time Data transportation and Ingestion. If I run the tutorial from my windows box, which is outside of Sandbox, (see [1]), I got the SocketTimeoutException (see [2]). I can see the socket connection is established from my windows box to HDP Sandbox and I can access Ambari and Hue from my windows box. I have verified that SELinux on HDP Sandbox has been disabled. However if I copy the jar from my windows box to HDP Sandbox and run it inside HDP Sandbox, everything is fine as I can see the messages have been successfully published. Could you please anyone give me a clue how to solve this issue? Many thanks in advance. [1] command to run Hortonwork Kafka Tutorial java -cp target/Tutorial-1.0-SNAPSHOT.jar com.hortonworks.tutorials.tutorial1.TruckEventsProducer sandbox.hortonworks.com:6667 sandbox.hortonworks.com:2181 [2] SocketTimeException 16/02/09 09:41:50 WARN client.ClientUtils$: Fetching topic metadata with correlation id 0 for topics [Set(truckevent)] from broker [BrokerEndPoint(0,sandbox.hortonworks.com,6667)] failed java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(Unknown Source) at sun.nio.ch.ChannelInputStream.read(Unknown Source) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Unknown Source) at kafka.utils.CoreUtils$.read(CoreUtils.scala:192) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:77) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$doSend(SyncProducer.scala:74) at kafka.producer.SyncProducer.send(SyncProducer.scala:115) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) at kafka.producer.async.DefaultEventHandler$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:68) at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:89) at kafka.utils.Logging$class.swallowError(Logging.scala:106) at kafka.utils.CoreUtils$.swallowError(CoreUtils.scala:51) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:68) at kafka.producer.Producer.send(Producer.scala:84) at kafka.javaapi.producer.Producer.send(Producer.scala:33) at com.hortonworks.tutorials.tutorial1.TruckEventsProducer.main(TruckEventsProducer.java:106) ... View more