@Zhen Zeng To create an HDF cluster with Cloudbreak, a KDC must be configured, unless you have registered an LDAP in Cloudbreak and select that when creating the cluster. During cluster creation, did you use test a KDC or an existing KDC? For configuring Cloudbreak to create a cluster that uses a KDC, please refer to the Enable Kerberos documentation for Cloudbreak 2.7. For complete instructions to create an HDF cluster with Cloudbreak, please refer to the Cloudbreak 2.7 documentation for Creating HDF Clusters. ... View more

I apologize that I couldn't think of a workaround, and that you'll have to set "Permission umask" for each processor. After NIFI-5575 is resolved, it will be included in a future HDF release, and you should be able to update your flow to remove the specific settings in each processor. ... View more

@Bob T Is the /usr/lib/hdinsight-datalake directory itself readable/executable by the user running NiFi? Without a specific FACL for the hdinsight-datalake directory, the user running NiFi needs to have read/execute permission on each dir in the path and read permission files in that dir to be able to access the JARs. I see the permissions on the JARs are wide open, but can you confirm read/execute on the directories? ... View more

@Bob T I think HdlAdiFileSystem was renamed in the version of hadoop-azure-datalake-2.7.3.2.6.5.8-7.jar you are using. Try updating the fs.adl.impl and fs.AbstractFileSystem.adl.impl values in core-site.xml: <property> <name>fs.adl.impl</name> <value>org.apache.hadoop.fs.adl.AdlFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.adl.impl</name> <value>org.apache.hadoop.fs.adl.Adl</value> </property> ... View more

@Bob T Could you please put stack traces inside of code blocks to make them a bit easier to read? It looks like you are still having classpath problems. Assuming that NiFi's lib directory is now restored to how it is from a "vanilla" install, I would check to make sure that you have the proper versions of the additional jars you're adding that work with Hadoop 2.7.3. That's the version of hadoop-client that is used by NiFi 1.5. It might help if you also (using code blocks) comment with a listing of the nifi/lib dir, the /usr/lib/hdinsight-datalake dir, and the contents of (or a link to) the xml files you've listed in "Hadoop Configuration Resources", sanitized of any information you don't want to post publicly. 🙂 ... View more

@Bob T The link you included has instructions to put those jars in /usr/lib/hdinsight-datalake, and then in the processor configuration for FetchHDFS, set the property "Additional Classpath Resources" to "/usr/lib/hdinsight-datalake". You don't have to use those specific directories, but they must be in a directory that NiFi can read, and NiFi needs to have read permissions on each jar. Also, please remove the jars you added to NiFi's lib directory. Adding jars directly to NiFi's lib directory can break NiFi itself, or some of its components. It has to do with how classloaders are created so that different NiFi components can use the versions of dependencies that they need. If jars are placed directly in NiFi's lib directory, it may override a dependency for a component and cause it to fail. Could you please perform those steps and try running the flow again? ... View more

@Saikrishna Tarapareddy, you may want to create a process group that contains the instantiation of your template, and then create connections from the areas of your flow to that process group. That way, you have one "instance" of the template created, and you'll only need to do your modifications once. You can always save a new template (for other instantiations, exporting, etc) with your modifications. I do admit that making connections across process groups in order to reuse a specific group may make the flow a bit harder to read, but eventually NiFi will support some improvements to make this easier/cleaner to do in a flow. ... View more

@Ian Neethling Could you provide a bit more about how you've installed NiFi? Are you running HDF, or just NiFi by itself? Did you install from an RPM, or did you install from a tar/gzip? ... View more