Envelope example packaging on Windows

[#1]

I maven "package"d the envelope project to generate the jar files in the build folder(and target folder of the external sources) using IntelliJ's Maven(v3.3.9) plugin on Window.I ran into below problems when trying out the filesystem example. Apparently, jar files created on Unix environment are working. We're on CDH15.5 and spark version 2.3.0.cloudera3. To create the jar files, I skipped tests.
 

[rxsubram@edpdev-b4bbb72c envelope-master]$ spark2-submit /opt/cloudera/scripts/envelope-master/*.jar examples/filesystem/filesystem.conf
18/12/04 22:25:31 INFO envelope.EnvelopeMain: Envelope application started
Exception in thread "main" com.typesafe.config.ConfigException$Parse: /opt/cloudera/scripts/envelope-master/envelope-hbase-0.6.0.jar: 2: Key '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"' may not be followed by token: '�0�|C' (if you intended '�0�|C' to be part of the value for '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"', try enclosing the value in double quotes)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:431)
at com.typesafe.config.impl.Parser$ParseContext.parseObject(Parser.java:786)
at com.typesafe.config.impl.Parser$ParseContext.parse(Parser.java:993)
at com.typesafe.config.impl.Parser.parse(Parser.java:34)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:222)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:210)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:176)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:170)
at com.typesafe.config.impl.Parseable.parse(Parseable.java:227)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:595)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:599)
at com.cloudera.labs.envelope.utils.ConfigUtils.configFromPath(ConfigUtils.java:38)
at com.cloudera.labs.envelope.EnvelopeMain.main(EnvelopeMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/12/04 22:25:32 INFO util.ShutdownHookManager: Shutdown hook called
18/12/04 22:25:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-dc509fa3-d861-424c-9327-3f17523cb232

I also observed that when I build envelope, both 1.0.2 and 1.2.1 jar versions of typesafe config are downloaded. However, I couldn't get the dependency tree for you. the lib module's build breaks (I'm on Windows without setting up the needed services, of course!) Do let me know if I can get this setup done somehow on Windows and I'd be happy to debug with you.

Warm Regards,

Rema S

[#2]

I maven "package"d the envelope project to generate the jar files in the build folder(and target folder of the external sources) using IntelliJ's Maven(v3.3.9) plugin on Window.I ran into below problems when trying out the filesystem example. Apparently, jar files created on Unix environment are working. We're on CDH15.5 and spark version 2.3.0.cloudera3. To create the jar files, I skipped tests.
 

[rxsubram@edpdev-b4bbb72c envelope-master]$ spark2-submit /opt/cloudera/scripts/envelope-master/*.jar examples/filesystem/filesystem.conf
18/12/04 22:25:31 INFO envelope.EnvelopeMain: Envelope application started
Exception in thread "main" com.typesafe.config.ConfigException$Parse: /opt/cloudera/scripts/envelope-master/envelope-hbase-0.6.0.jar: 2: Key '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"' may not be followed by token: '�0�|C' (if you intended '�0�|C' to be part of the value for '"PK\u0003\u0004\u0014\u0000\b\b\b\u0000���M\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014\u0000\u0004\u0000META-INF/MANIFEST"."MF��\u0000\u0000E��"', try enclosing the value in double quotes)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:431)
at com.typesafe.config.impl.Parser$ParseContext.parseObject(Parser.java:786)
at com.typesafe.config.impl.Parser$ParseContext.parse(Parser.java:993)
at com.typesafe.config.impl.Parser.parse(Parser.java:34)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:222)
at com.typesafe.config.impl.Parseable.rawParseValue(Parseable.java:210)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:176)
at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:170)
at com.typesafe.config.impl.Parseable.parse(Parseable.java:227)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:595)
at com.typesafe.config.ConfigFactory.parseFile(ConfigFactory.java:599)
at com.cloudera.labs.envelope.utils.ConfigUtils.configFromPath(ConfigUtils.java:38)
at com.cloudera.labs.envelope.EnvelopeMain.main(EnvelopeMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/12/04 22:25:32 INFO util.ShutdownHookManager: Shutdown hook called
18/12/04 22:25:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-dc509fa3-d861-424c-9327-3f17523cb232

I also observed that when I build envelope, both 1.0.2 and 1.2.1 jar versions of typesafe config are downloaded. However, I couldn't get the dependency tree for you. the lib module's build breaks (I'm on Windows without setting up the needed services, of course!) Do let me know if I can get this setup done somehow on Windows and I'd be happy to debug with you.

Warm Regards,

Rema S

[#3 · Accepted solution]

Hi Rema,

It looks like the "/opt/cloudera/scripts/envelope-master/*.jar" argument is expanding out to multiple jar files, and then Envelope is assuming that the second jar is the configuration file (you can see the "PK" in the exception -- that's the start of a jar file). I'd recommend specifying the single Envelope jar instead of using the wildcard.

Jeremy

[#4]

Yes, that worked. I kept thinking the configuration file was the problem. I tried the *.jar because the readme command didn't work.

(**Run the Envelope job**

spark2-submit build/envelope/target/envelope-*.jar examples/filesystem/filesystem.conf). thanks for the help, Jeremy!

Warm Regards,

Rema S

[#5]

I'm happy to see you resolved the issue @angoothachap. If @Jeremy Beard 's reply was the solution, you could mark it as such to make it easier for others to find an answer in the future. :)