Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Exception in thread "main" java.lang.RuntimeException: core-site.xml not found

Exception in thread "main" java.lang.RuntimeException: core-site.xml not found

Explorer

Hi,

 

    I'm trying to run example scala project in CDSW. When I try to put data to hdfs I have and exception:

Exception in thread "main" java.lang.RuntimeException: core-site.xml not found

 

    I printed HADOOP_CONF_DIR in my scala code and it looks good:

 

println(sys.env("HADOOP_CONF_DIR"))
/etc/hadoop/conf:/etc/hive/conf

 

    Files exist in docker container.

 

   Spark also see configration files. Lines return proper output in my opinion :

 

import org.apache.hadoop.fs._
import org.apache.hadoop.conf._
import scala.collection.JavaConverters._

val fs = FileSystem.get(sc.hadoopConfiguration)
fs.getConf.iterator().asScala.foreach(println)

 

 

    I'm using CDSW 1.5.0 and CM 6.2.0.

 

    Any advices why spark does not see core-site.xml?

 

Regards,

 

Bart