Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues post hdp 3.1 migration

Highlighted

Issues post hdp 3.1 migration

New Contributor

We have used union all to join 2 data sets with different datatypes. It was working before hdp 3.1 migration. But after migration this is failing with Unsupported operation exception. I completely understand datatypes should be matching for Union or Union ALL. But how it was working before migration. Even after migration if we are writing the result from Driver, it is working. If we are caching intermediate result, it is working. But if we are writing the result set outside driver class, it is not working. I didn't understand the difference.

1 REPLY 1
Highlighted

Re: Issues post hdp 3.1 migration

New Contributor

Please find the below error we are getting

{"level":"ERROR","timestamp":"2020-03-06 03:18:50,038","thread":"Executor task launch worker for task 14","className":"Logging.scala", "line":"91","ApplicationInfo":Exception in task 0.0 in stage 13.0 (TID 14)}
java.lang.UnsupportedOperationException
at org.apache.spark.sql.vectorized.ArrowColumnVector$ArrowVectorAccessor.getLong(ArrowColumnVector.java:215)
at org.apache.spark.sql.vectorized.ArrowColumnVector.getLong(ArrowColumnVector.java:89)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:148)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Don't have an account?
Coming from Hortonworks? Activate your account here