Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Writestream or write in hive using spark-submit (spark 2.3.2)

Highlighted

Writestream or write in hive using spark-submit (spark 2.3.2)

Explorer

Hello All, 

 

I have a strange error when I run my code with spark-submit but the code work really fine on intellj or zeppelin. The code seem to failed at the step of writestream in Hive. I have already test writestream in console and it work fine but when it is writestream (or a write) in Hive  we obtain this error : 

Exception in thread "main" java.lang.IllegalStateException: createWriterFactory should not be called with SupportsWriteInternalRow.
at org.apache.spark.sql.sources.v2.writer.SupportsWriteInternalRow.createWriterFactory(SupportsWriteInternalRow.java:36)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:55)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)

 

Has anyone ever encountered this error or knows what's wrong.

Thanks in advance.

Don't have an account?
Coming from Hortonworks? Activate your account here