Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark classpath issue when using the "java" command

Highlighted

Spark classpath issue when using the "java" command

Explorer

Hi...

 

We run a Spark jiob that uses morphlines and pass the dependencies via the DistributedCache. The code works as expected, apart from cases where a HOCON "java" command is used to run a few lines of Java code from within the morphline file.

 

Exactly the same morphlien works fine in a local environment (outside Spark), which probably denotes a classpath or similar issue.

 

The exceptions thrown by the Spark worker are pretty clear about it:

* package com.typesafe.config does not exist

* package org.kitesdk.morphline.api does not exist

* package <whatever used in the "import" statement> does not exist

 

Any ideas?

 

Thanks.

 

 

4 REPLIES 4
Highlighted

Re: Spark classpath issue when using the "java" command

Expert Contributor
What version of CDH is this? Try using crunchindexertool CLI, or compare how you are passing dependencies to how crunchindexertool passes dependencies. If this is your own job code you might need this: https://github.com/cloudera/search/blob/cdh5-1.0.0_5.3.2/search-crunch/src/main/java/org/apache/solr...

Wolfgang


Highlighted

Re: Spark classpath issue when using the "java" command

Explorer

Thanks for the hints.

 

We are using CDH 5.1.2.

 

Even if we fix the classpath, it will still require a Java compiler in the installation site for any morphline that uses the "java" command, correct?

 

Re: Spark classpath issue when using the "java" command

Expert Contributor
Yes, it needs a compiler and the JDK includes a compiler.

Highlighted

Re: Spark classpath issue when using the "java" command

Explorer

Yes, we were thinking of ways to remove that requirement or make it configurable (like a pluggable compiler), but it does not seem possible.

 

Maybe it is time you guys introduced a "python" command. Maybe we should replicate the "java' command with Jython :-)

 

Don't have an account?
Coming from Hortonworks? Activate your account here