Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How does Livy determine AbsolutePath vs RelativePath?

How does Livy determine AbsolutePath vs RelativePath?

New Contributor
Hi,
 
    I am trying to figure out how Livy determines the path to the jars. So far seems if it is run in local mode, it adds prefix "file://". If it runs in Yarn mode, it adds prefix "hdfs://".
 
    I am reading the code and found in file SparkSubmitProcessBuilder.scala, it has the following function:
private def fromPath(path: Path) = path match {
case AbsolutePath(p) => p
case RelativePath(p) =>
if (p.startsWith("hdfs://")) {
p
} else {
fsRoot + "/" + p
}
}
 
    However, I just cannot find how AbsolutePath or RelativePath is defined. At the beginning of the file it has the following and then I cannot find any definition of the function or class.
sealed trait Path
case class AbsolutePath(path: String) extends Path
case class RelativePath(path: String) extends Path
 
    I am quite new to scala so I would love to understand the magic here. Essentially, how does Livy determine if a given path is an absolute path or relative path here?
 
Thanks
Lin
 
 
 
 
 
 
Don't have an account?
Coming from Hortonworks? Activate your account here