Pushing to ADLS using Nifi

[#1]

Nifi build on HDF 3.1.2 and HDF 3.1.0 fail with a depency issue when trying to push data to ADLS. This is because the new version of ADLS has some dependency on hadoop 2.8 feature, which is not available in 2.7.3 which is referenced by nifi.

to fix this you can build nifi again. You could eith build it against hadoop 2.8 or againt hdp 2.6.x which should have the classes that ADLS depends on.

to do that

git clone nifi repository
cd <nif-repo-home>/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/
vi pom.xml

add a hadoop.version property to the pom.xml as shown below. if already set, no change is needed.

change nifi version to match the nifi version you are running for the parent

<parent>

        <groupId>org.apache.nifi</groupId>

        <artifactId>nifi-hadoop-libraries-bundle</artifactId>

        <version>1.5.0.3.1.2.0-7</version>

    </parent>

    <artifactId>nifi-hadoop-libraries-nar</artifactId>

    <packaging>nar</packaging>

    <properties>

        <maven.javadoc.skip>true</maven.javadoc.skip>

        <source.skip>true</source.skip>

        <curator.version>2.11.0</curator.version>

        <hadoop.version>2.7.3</hadoop.version>

    </properties>

cd ..
vi pom.xml

change the nifi-nar-bunlde version to match your nifi version as shown below

<parent>
        <groupId>org.apache.nifi</groupId>

        <artifactId>nifi-nar-bundles</artifactId>

        <version>1.5.0.3.1.2.0-7</version>

    </parent>

run the maven build using the command below. change hadoop.version to match your version of hadoop. the nar will avaliable under nifi-hadoop-libraries-nar/target folder. Take that nar and replace the existing nar under nifi/lib.

mvn clean package -Dhadoop.version=2.7.3.2.6.5.0-292

Ensure you have the right jars for adls downloaded into a folder accesible to nifi. Add the folder path to Additional classpath option in PUTHDFS. for hdp 2.6.x you can find the needed jars under /usr/hdp/2.6.x..../hadoop/

you will need the following jars.

azure-data-lake-store-sdk-2.2.5.jar
hadoop-azure-2.7.3.2.6.5.0-292.jar
hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar

start nifi and push files , hope it works.