@Lenu K Are you using a keytab with PutHDFS? You could set the permission of the directory to which PutHDFS is writing to allow group/other read access so that Hive can read that directory, and also set the umask in the PutHDFS processor to write files that are readable by group/other. ... View more

@Daniel Niguse Going by the tags on this question, it looks like you may be using HDP 2.6.0 version of Hortonworks Sandbox? You'll want to check to see if docker is forwarding port 9090 to the container. The HDP 2.6.5 version of Hortonworks Sandbox looks like it forwards that port by default. Sandbox Port Forwards - HDP 2.6.5 I would also suggest making modifications to the NiFi configuration through Ambari, rather than directly modifying the configuration files. The "nifi-ambari-config" config-site contains the properties for the HTTP(S) ports. ... View more

@Zhen Zeng To create an HDF cluster with Cloudbreak, a KDC must be configured, unless you have registered an LDAP in Cloudbreak and select that when creating the cluster. During cluster creation, did you use test a KDC or an existing KDC? For configuring Cloudbreak to create a cluster that uses a KDC, please refer to the Enable Kerberos documentation for Cloudbreak 2.7. For complete instructions to create an HDF cluster with Cloudbreak, please refer to the Cloudbreak 2.7 documentation for Creating HDF Clusters. ... View more

I apologize that I couldn't think of a workaround, and that you'll have to set "Permission umask" for each processor. After NIFI-5575 is resolved, it will be included in a future HDF release, and you should be able to update your flow to remove the specific settings in each processor. ... View more

@Alaa Nabil From the information you've provided, it looks like PutHDFS should work. Without seeing your nifi-app.log, core-site.xml, and hdfs-site.xml files, I am not sure what is keeping PutHDFS from being able to write files to HDFS. Does this happen for every file sent to PutHDFS? You could run through the checklist in this StackOverflow post, as well. ... View more

@Alaa Nabil A few questions for you: Are there 0-byte files in HDFS that correspond to the files you're trying to send with PutHDFS? This would mean that PutHDFS was able to create the file when contacting the namenode, but may not be able to reach the datanode. Are you able to use the HDFS command line client to send files to HDFS from the same node on which NiFi is running? Are you running the HDP cluster in a VM, or the Hortonworks Sandbox? There are ports need to be open on the hosts that are datanodes. Port 50010 may not be open, making the datanode unreachable by NiFi. You can see the default ports here: https://ambari.apache.org/1.2.3/installing-hadoop-using-ambari/content/reference_chap2_1.html ... View more

@Bob T Is the /usr/lib/hdinsight-datalake directory itself readable/executable by the user running NiFi? Without a specific FACL for the hdinsight-datalake directory, the user running NiFi needs to have read/execute permission on each dir in the path and read permission files in that dir to be able to access the JARs. I see the permissions on the JARs are wide open, but can you confirm read/execute on the directories? ... View more

@Bob T I think HdlAdiFileSystem was renamed in the version of hadoop-azure-datalake-2.7.3.2.6.5.8-7.jar you are using. Try updating the fs.adl.impl and fs.AbstractFileSystem.adl.impl values in core-site.xml: <property> <name>fs.adl.impl</name> <value>org.apache.hadoop.fs.adl.AdlFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.adl.impl</name> <value>org.apache.hadoop.fs.adl.Adl</value> </property> ... View more

@Bob T Could you please put stack traces inside of code blocks to make them a bit easier to read? It looks like you are still having classpath problems. Assuming that NiFi's lib directory is now restored to how it is from a "vanilla" install, I would check to make sure that you have the proper versions of the additional jars you're adding that work with Hadoop 2.7.3. That's the version of hadoop-client that is used by NiFi 1.5. It might help if you also (using code blocks) comment with a listing of the nifi/lib dir, the /usr/lib/hdinsight-datalake dir, and the contents of (or a link to) the xml files you've listed in "Hadoop Configuration Resources", sanitized of any information you don't want to post publicly. 🙂 ... View more