Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Trying to write a df as orc to hdfs of 3.5 GB size using spark but just hangs!!!

Trying to write a df as orc to hdfs of 3.5 GB size using spark but just hangs!!!

New Contributor

Hi All, I have a dataframe of size 3.5GB in orc format and the columns,records are 383 and 13145532 respectively. I am trying to read the data which I could and I can get the count and show(50) is working but when I trying to write to hdfs as an orc without any transformation it just hangs there for hours..PFB my spark -submit command: spark-submit --class com.test.new.castingadhoc.castingdata --master yarn --executor-memory 30g --num-executors 10 --executor-cores 7 --conf spark.local.dir=/newlog/log/spark_dir --conf spark.debug.maxToStringFields=400 --conf spark.default.parallelism=56 --conf spark.shuffle.partitions=56 --conf spark.executor.memoryOverhead=5G /samplest/codesample-1.0-SNAPSHOT.jar ; The spark version:2.2

Don't have an account?
Coming from Hortonworks? Activate your account here