Hive Maven Error in HDP 3.1.4

ask_bill_brooks · ‎12-10-2019

Hi,

We're using the Hortonworks repository to build our Hadoop project using Maven. For our project, one of the dependencies is on hive-exec JAR file. The version in HDP 3.1.4 for hive-exec in our HDP distribution is 3.1.0.3.1.4.0-315 and the dependency is specified as:

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>3.1.0.3.1.4.0-315</version>
        </dependency>

As Maven resolves all transitive dependencies automatically, this JAR file is giving the following error:

Could not find artifact org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.7.3.2.6.5.0-SNAPSHOT in hortonworks (http://repo.hortonworks.com/content/groups/public)

By viewing the dependency tree for hive-exec, this specific snippet is related to the error:

org.apache.hive:hive-exec:jar:3.1.0.3.1.4.0-315:provided
   org.apache.hive:hive-shims:jar:3.1.0.3.1.4.0-315:provided
          org.apache.hive.shims:hive-shims-common:jar:3.1.0.3.1.4.0-315:provided
                org.apache.hive.shims:hive-shims-0.20:jar:3.1.0.3.1.4.0-315:provided
                      org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.7.3.2.6.5.0-SNAPSHOT:provided

The version 2.7.3.2.6.5.0-SNAPSHOT for hadoop-yarn-server-resourcemanager is specified in the parent POM of the Hive project within the Hortonworks repository. From the error, and from own research, this SNAPSHOT version does not exist in any repository, Hortonworks, Maven Central or Apache-Snapshot repository which leads to the build error. The version number is obtained from the property defined in the Apache Hive parent pom:

<hadoop2.version>2.7.3.2.6.5.0-SNAPSHOT</hadoop2.version>

Prior version of HDP (3.1.0), hive-exec had a dependency on hive-shims-0.23 with the dependency tree of:

org.apache.hive:hive-exec:jar:3.1.0.3.1.0.0-78:provided
   org.apache.hive:hive-shims:jar:3.1.0.3.1.0.0-78:provided
          org.apache.hive.shims:hive-shims-common:jar:3.1.0.3.1.0.0-78:provided
                org.apache.hive.shims:hive-shims-0.23:jar:33.1.0.3.1.0.0-78:provided
                      org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.1.1.3.1.0.0-78:provided

It looks like hive-shims version was lowered in HDP 3.1.4 and the hadoop-yarn-server-resourcemanager versions specified does not exist. Since the transitive dependencies are not resolved as per what's provided in the POM files for the projects/dependencies in the Hortonworks repostiory, is this an issue with HDP 3.1.4? Or is there something I can do in my own POM file to resolve this?

Thanks.

usd84 · ‎12-11-2019

I was able to get around this by excluding the transitive dependencies since I only needed the hive-exec file to compile the project. However, I still think that the project file should be fixed in the Hortonworks repo as it's pointing to an artifact that does not exist in any public repositories.