Reply
New Contributor
Posts: 3
Registered: ‎01-23-2017

HADOOP_CONF_DIR is set to a folder list, seems different from Apache spark

On both the quickstart VM and our cloudera 5.5 cluster, I find that HADOOP_CONF_DIR as set by spark-submit is set to a folder list.  On our 5.5 cluster, it is set to:

/opt/cloudera/parcels/CDH-5.5.5-1.cdh5.5.5.p0.3/lib/spark/conf/yarn-conf:/etc/hive/conf

This seems different from Apache spark.

 

Please explain.

 

Highlighted
Champion
Posts: 563
Registered: ‎05-16-2016

Re: HADOOP_CONF_DIR is set to a folder list, seems different from Apache spark

A parcel is a binary distribution format containing the program files, along with additional metadata used by Cloudera Manager, The reason it is different from Apache spark is because your quickstart vmis using parcels .

This should not stop you from firing spark jobs. Let me know if this is suffice . 

Announcements