Cloudera Community

Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!
subbiramreddyp
user rank
Explorer

Re: Save dataframe with header in spark 1.6

by Explorer in Support Questions
‎04-02-2018 06:01 AM
‎04-02-2018 06:01 AM
@Rahul Soni Thanks for the response Rahul, I have doubt regarding spark version in certification are they updated the spark version to 2.xx or still they provide 1.6 only. ... View more

Save dataframe with header in spark 1.6

by Explorer in Support Questions
‎03-29-2018 09:56 AM
‎03-29-2018 09:56 AM
Hi I am going to write HPCD spark certification in that I need to save dataframe to HDFS Doubts: 1.Do I need to store data with header or not If need to store any approach please suggest me because they are not going to give any databricks package without header I can. Thanks in Advance Subbiram ... View more
Labels:

Kafka spark structuredstreaming

by Explorer in Support Questions
‎02-22-2018 05:14 AM
‎02-22-2018 05:14 AM
Hi, we are trying to implement kafka with spark structuredstreaming for that we imported the spark-sql-kafka-0-10_2.11-2.0.2 module using spark-shell while executing it shows below error: pplyOrElse(StreamExecution.scala:153) at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$4.apply(TreeNode.scala:306) at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187) at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256) at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan$lzycompute(StreamExecution.scala:153) at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan(StreamExecution.scala:147) at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$runBatches(StreamExecution.scala:276) at org.apache.spark.sql.execution.streaming.StreamExecution$anon$1.run(StreamExecution.scala:206) Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArrayDeserializer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 47 more Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: org/apache/kafka/common/serialization/ByteArrayDeserializer === Streaming Query === Identifier: [id = f230ea99-b84b-44ef-ba49-e8ecb02ae810, runId = 82357c6a-306d-4e6f-96a8-6238302ff59d] Current Committed Offsets: {} Current Available Offsets: {} Current State: INITIALIZING Thread State: RUNNABLE at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$runBatches(StreamExecution.scala:343) at org.apache.spark.sql.execution.streaming.StreamExecution$anon$1.run(StreamExecution.scala:206) Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:74) at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:246) even when we running example using ./runexample command we are facing the same issue. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎06-19-2018 05:11 AM
Community Statistics
Member Since ‎02-22-2018 05:03 AM
Last Visited ‎06-19-2018 05:11 AM
Posts 13
Kudos received 1