Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Introduce a new file system in Livy

Introduce a new file system in Livy

New Contributor

Currently, seems Livy has hardcoded the file path to be hdfs if running against yarn mode according to the following code.

 

  def filesystemRoot(): String = {
    Configuration cE = new HdfsConfiguration(false);
    sessionKind() match {
      case Process() => "file://"
      case Yarn() => "hdfs://"
    }
  }

 

We would like to use wasb:// as default but want to make this generic so it can be easily extended to even other FS. We would like to seek advice on how this should be done.

 

Option 1:

Hadoop has a core-site.xml which lives in the HADOOP_CONF_DIR. It has a deafultFS setting which we can use here.

 

Option 2:

Introduce a new config value in livy-defaults.conf (say livy.yarn.fs) which points to the default fs to be used. We can default it to hdfs if not configured.

 

Which is a more preferrable option?

 

Thanks

Lin