We run a Spark jiob that uses morphlines and pass the dependencies via the DistributedCache. The code works as expected, apart from cases where a HOCON "java" command is used to run a few lines of Java code from within the morphline file.
Exactly the same morphlien works fine in a local environment (outside Spark), which probably denotes a classpath or similar issue.
The exceptions thrown by the Spark worker are pretty clear about it:
* package com.typesafe.config does not exist
* package org.kitesdk.morphline.api does not exist
* package <whatever used in the "import" statement> does not exist
Thanks for the hints.
We are using CDH 5.1.2.
Even if we fix the classpath, it will still require a Java compiler in the installation site for any morphline that uses the "java" command, correct?
Yes, we were thinking of ways to remove that requirement or make it configurable (like a pluggable compiler), but it does not seem possible.
Maybe it is time you guys introduced a "python" command. Maybe we should replicate the "java' command with Jython :-)