Reply
Explorer
Posts: 6
Registered: ‎09-26-2018
Accepted Solution

Envelope example packaging on Windows

[ Edited ]

I maven "package"d the envelope project to generate the jar files in the build folder(and target folder of the external sources) using IntelliJ's Maven(v3.3.9) plugin on Window.I ran into below problems when trying out the filesystem example. Apparently, jar files created on Unix environment are working. We're on CDH15.5 and spark version 2.3.0.cloudera3. To create the jar files, I skipped tests.
 

 

[rxsubram@edpdev-b4bbb72c envelope-master]$ spark2-submit /opt/cloudera/scripts/envelope-master/*.jar examples/filesystem/filesystem.conf
18/12/04 22:25:31 INFO envelope.EnvelopeMain: Envelope application started
Exception in thread "main" com.typesafe.config.ConfigException$Parse: /opt/cloudera/scripts/envelope-master/envelope-hbase-0.6.0.jar: 2: Key '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"' may not be followed by token: '�0�|C' (if you intended '�0�|C' to be part of the value for '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"', try enclosing the value in double quotes)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:431)
at com.typesafe.config.impl.Parser$ParseContext.parseObject(Parser.java:786)
at com.typesafe.config.impl.Parser$ParseContext.parse(Parser.java:993)
at com.typesafe.config.impl.Parser.parse(Parser.java:34)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:222)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:210)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:176)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:170)
at com.typesafe.config.impl.Parseable.parse(Parseable.java:227)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:595)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:599)
at com.cloudera.labs.envelope.utils.ConfigUtils.configFromPath(ConfigUtils.java:38)
at com.cloudera.labs.envelope.EnvelopeMain.main(EnvelopeMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/12/04 22:25:32 INFO util.ShutdownHookManager: Shutdown hook called
18/12/04 22:25:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-dc509fa3-d861-424c-9327-3f17523cb232

 

 

I also observed that when I build envelope, both 1.0.2 and 1.2.1 jar versions of typesafe config are downloaded. However, I couldn't get the dependency tree for you. the lib module's build breaks (I'm on Windows without setting up the needed services, of course!) Do let me know if I can get this setup done somehow on Windows and I'd be happy to debug with you.

 

Warm Regards,

Rema S

Cloudera Employee
Posts: 34
Registered: ‎08-26-2015

Re: Envelope example packaging on Windows

Hi Rema,

 

It looks like the "/opt/cloudera/scripts/envelope-master/*.jar" argument is expanding out to multiple jar files, and then Envelope is assuming that the second jar is the configuration file (you can see the "PK" in the exception -- that's the start of a jar file). I'd recommend specifying the single Envelope jar instead of using the wildcard.

 

Jeremy

Explorer
Posts: 6
Registered: ‎09-26-2018

Re: Envelope example packaging on Windows

Yes, that worked. I kept thinking the configuration file was the problem. I tried the *.jar because the readme command didn't work.

(**Run the Envelope job**

spark2-submit build/envelope/target/envelope-*.jar examples/filesystem/filesystem.conf). thanks for the help, Jeremy!

 

Warm Regards,

Rema S

Highlighted
Posts: 877
Kudos: 102
Solutions: 54
Registered: ‎04-06-2015

Re: Envelope example packaging on Windows

I'm happy to see you resolved the issue @angoothachap. If @Jeremy Beard 's reply was the solution, you could mark it as such to make it easier for others to find an answer in the future. :) 

 

Screen Shot 2018-12-05 at 4.16.56 PM.png



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB