Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spring Boot Hive Warehouse Connector HDP 3.11

Highlighted

Spring Boot Hive Warehouse Connector HDP 3.11

New Contributor

Is there a way to use the hive-warehouse-connector with spring boot? It seems like I need that to connect spark and hive in hdp version 3+. I get the error below when I try to start my jar.

Exception in thread "main" java.lang.IllegalStateException: Failed to get nested archive for entry BOOT-INF/lib/hive-warehouse-connector_2.11-1.0.0.3.1.0.0-78.jar

        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:108)

        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchives(JarFileArchive.java:86)

        at org.springframework.boot.loader.ExecutableArchiveLauncher.getClassPathArchives(ExecutableArchiveLauncher.java:70)

        at org.springframework.boot.loader.Launcher.launch(Launcher.java:49)

        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)

Caused by: java.io.IOException: Unable to open nested jar file 'BOOT-INF/lib/hive-warehouse-connector_2.11-1.0.0.3.1.0.0-78.jar'

        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:256)

        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:241)

        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:103)

        ... 4 more

Caused by: java.lang.IllegalStateException: Zip64 archives are not supported

        at org.springframework.boot.loader.jar.CentralDirectoryEndRecord.getNumberOfRecords(CentralDirectoryEndRecord.java:124)

        at org.springframework.boot.loader.jar.JarFileEntries.visitStart(JarFileEntries.java:120)

        at org.springframework.boot.loader.jar.CentralDirectoryParser.visitStart(CentralDirectoryParser.java:89)

        at org.springframework.boot.loader.jar.CentralDirectoryParser.parse(CentralDirectoryParser.java:57)

        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:123)

        at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:111)

        at org.springframework.boot.loader.jar.JarFile.createJarFileFromFileEntry(JarFile.java:289)

        at org.springframework.boot.loader.jar.JarFile.createJarFileFromEntry(JarFile.java:264)

        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:252)

        ... 6 more


I am running it as a regular jar since I have my spark configured in the spring boot configuration and can leverage it that way.


What my goal of this application is to run the jar in a server within the hadoop cluster, sftp some csv to the local server, copy them to hdfs then run the either run spark jobs to read the files and insert to hive or just run a hive load command to insert the csv directly.

This is the dependency that is giving me the problem.

<dependency>
   <groupId>com.hortonworks.hive</groupId>
   <artifactId>hive-warehouse-connector_${scala.version}</artifactId>
   <version>1.0.0.3.1.0.0-78</version>
</dependency>