Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Expert Contributor
Nifi build on HDF 3.1.2 and HDF 3.1.0 fail with a depency issue when trying to push data to ADLS. This is because the new version of ADLS has some dependency on hadoop 2.8 feature, which is not available in 2.7.3 which is referenced by nifi.

to fix this you can build nifi again. You could eith build it against hadoop 2.8 or againt hdp 2.6.x which should have the classes that ADLS depends on.

to do that

git clone nifi repository
cd <nif-repo-home>/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/
vi pom.xml

add a hadoop.version property to the pom.xml as shown below. if already set, no change is needed.

change nifi version to match the nifi version you are running for the parent

<parent>

        <groupId>org.apache.nifi</groupId>

        <artifactId>nifi-hadoop-libraries-bundle</artifactId>

        <version>1.5.0.3.1.2.0-7</version>

    </parent>

    <artifactId>nifi-hadoop-libraries-nar</artifactId>

    <packaging>nar</packaging>

    <properties>

        <maven.javadoc.skip>true</maven.javadoc.skip>

        <source.skip>true</source.skip>

        <curator.version>2.11.0</curator.version>

        <hadoop.version>2.7.3</hadoop.version>

    </properties>
cd ..
vi pom.xml

change the nifi-nar-bunlde version to match your nifi version as shown below

<parent>
        <groupId>org.apache.nifi</groupId>

        <artifactId>nifi-nar-bundles</artifactId>

        <version>1.5.0.3.1.2.0-7</version>

    </parent>

run the maven build using the command below. change hadoop.version to match your version of hadoop. the nar will avaliable under nifi-hadoop-libraries-nar/target folder. Take that nar and replace the existing nar under nifi/lib.

mvn clean package -Dhadoop.version=2.7.3.2.6.5.0-292

Ensure you have the right jars for adls downloaded into a folder accesible to nifi. Add the folder path to Additional classpath option in PUTHDFS. for hdp 2.6.x you can find the needed jars under /usr/hdp/2.6.x..../hadoop/

you will need the following jars.

azure-data-lake-store-sdk-2.2.5.jar
hadoop-azure-2.7.3.2.6.5.0-292.jar
hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar

start nifi and push files , hope it works.

447 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎08-07-2018 05:23 PM
Updated by:
 
Contributors
Top Kudoed Authors